Fast binary serialization (save_binary / load_binary)

Goal

In addition to the standard pickle support already available on lightsim2grid.network.LSGrid (see Elements modeled), lightsim2grid offers a second, additive serialization path: save_binary / load_binary.

This is not a replacement for pickle. It is a much faster, custom binary format meant for the case where you repeatedly re-load the same grid on the same machine / lightsim2grid build and care more about speed than about portability. Pickle remains the way to go if you need to move a grid across python versions, machines, or lightsim2grid versions, or if you need it to work with multiprocessing / ray / etc.

It is available on lightsim2grid.network.LSGrid itself, as well as on every individual element container that has its own state (LoadContainer, LineContainer, GeneratorContainer, …), exactly mirroring what is already picklable:

import grid2op
from lightsim2grid import LightSimBackend

env = grid2op.make(..., backend=LightSimBackend())
grid = env.backend._grid

# save the whole grid to a fast binary file
grid.save_binary("my_grid.lsb")

# ... later, possibly in another python process on the same machine ...
from lightsim2grid.network import LSGrid
grid_reloaded = LSGrid.load_binary("my_grid.lsb")

Individual element containers work the same way:

loads = grid.get_loads()
loads.save_binary("my_loads.lsb")

from lightsim2grid.elements import LoadContainer
loads_reloaded = LoadContainer.load_binary("my_loads.lsb")

Note

Version compatibility is decided by a binary format version (an integer stored in the file header, see BINARY_FORMAT_VERSION in src/core/BinaryArchive.hpp), not by the lightsim2grid version: the format number is only bumped when the serialized layout actually changes, so all lightsim2grid versions sharing the same format number can read each other’s files. Loading a file with an unsupported format number raises a RuntimeError naming both versions and both format numbers.

Warning

load_binary validates its input and always raises a RuntimeError (rather than crashing, exhausting memory, or silently returning garbage) on ill-formed files: garbage or truncated files, corrupted internal sizes (every count in the file is checked against the real file size before anything is allocated), files that contain an object of a different type (eg loading a LoadContainer file with StorageContainer.load_binary), and files with unexpected trailing bytes are all rejected. On top of that byte-level validation, loading a whole grid also runs lightsim2grid.network.LSGrid.check_grid() (the same consistency check used everywhere a grid is loaded): a file that is byte-wise well-formed but carries an out-of-range index (bus / substation / topology-vector) is rejected with an IndexError (out-of-range index) or a RuntimeError (structural inconsistency) rather than being loaded into a state that would fault on the next powerflow. save_binary on the other hand is atomic by default: it writes to a temporary file that only replaces the destination once fully written, so an interrupted save never destroys a previously saved file. Pass atomic=False to write the destination directly instead (marginally faster – it skips one temporary file and rename – but without that protection).

Note

The solver is saved by name. A grid records the registry name of the AC and DC algorithm it was using ('NR_SparseLU', or whatever a plugin registered itself as), and load_binary re-selects the solver with that name. Two consequences:

• if that solver is not registered when you load the file – a plugin you have not loaded in this process, or a built-in needing an optional backend (KLU / NICSLU / CKTSO) this build lacks – load_binary raises, naming the solver and how to get it. Use LSGrid.load_binary_without_algorithm(path) to load the grid data and keep the default solvers instead, then pick one with change_algorithm;

• a name is not a strong identity. If two different plugins register the same solver name, the one loaded when the file is read is the one you get – it may be a completely different algorithm from the one that was used when the file was written, with no way for lightsim2grid to notice. Give your plugin solvers distinctive names (a project prefix, for instance) if you exchange files between environments.

Danger

Not even pickle is a portable format, and this binary format is even less so. Two rules to follow:

• Never load a binary file from a source you do not trust. load_binary rejects structurally ill-formed input and, via check_grid, a grid with out-of-range indices (see above), but a well-formed, internally-consistent file can still carry adversarial values – loading it means trusting its content, the same way loading a pickle file does. See also Security and trust boundaries.

• Generate the file on the machine that will consume it, with the same lightsim2grid version. The format stores raw native data (same endianness, real_type, …) with no checksum and no cross-platform migration; the binary-format-version check (see above) only catches layout changes the lightsim2grid authors explicitly bumped it for, not every possible mismatch. Use pickle instead if you need to move a grid across machines, python versions, or lightsim2grid versions.

Note

The binary format walks the exact same internal state (get_state / set_state) that pickle uses, so the two stay structurally in sync: anything that can be pickled can be saved with save_binary. See lightsim2grid/tests/test_binary_serialization.py for the full test suite (round trip, AC/DC powerflow round trip, format mismatch, cross-version load, corrupted / truncated / wrong-type files, atomicity, and a committed reference file guarding the layout against accidental changes).

Benchmarks

Here are some benchmarks made with:

• date: 2026-10-12 19:55 CEST

• system: Linux 6.8.0-60-generic

• OS: ubuntu 22.04

• processor: 13th Gen Intel(R) Core(TM) i7-13700H

• python version: 3.12.8.final.0 (64 bit)

• numpy version: 2.4.6

• pandas version: 2.3.3

• pandapower version: 3.5.4

• grid2op version: 1.12.5.dev0

• lightsim2grid version: 1.2.0

• lightsim2grid extra information:

  • klu_solver_available: True

  • nicslu_solver_available: True

  • cktso_solver_available: True

  • compiled_march_native: True

  • compiled_o3_optim: True

This benchmark is available by running, from the root of the lightsim2grid repository:

cd benchmarks
python3 benchmark_binary_serialization.py

Single grid (l2rpn_idf_2023)

A first, direct comparison on a single mid-sized grid (the same fixture used by the test suite):

l2rpn_idf_2023 (20 repeats)	time (ms/call)	speedup vs pickle	file size
save_binary (write)	0.421	1.2x	34444 bytes
load_binary (read)	0.337	1.3x
pickle.dump (write)	0.487	1.0x (reference)	35639 bytes
pickle.load (read)	0.448	1.0x (reference)
init_from_pandapower (case118, reference)	1.362
init_from_pypowsybl (ieee118, reference)	26.892

The “speedup vs pickle” column above is now computed and printed directly by benchmark_binary_serialization.py (previously it was a hand-written sentence below the table, computed separately from the same two numbers – a second place the ratio could have drifted out of sync with the table above it). save_binary / load_binary also produce a slightly smaller file than pickle on this grid.

Grid size scan

The benchmark script also scans a range of grid sizes (the same pandapower cases used in Benchmarks (grid size), from 14 to ~9241 buses), each LSGrid being built directly with lightsim2grid.network.init_from_pandapower(). In addition to save_binary / load_binary versus pickle, it also reports, as a reference, the time to read the original pandapower json file from disk and build an LSGrid from it from scratch (pp.from_json + init_from_pandapower) – this is the “no snapshot at all” baseline:

grid	nb bus	save_binary (ms)	pickle.dump (ms)	write speedup	load_binary (ms)	pickle.load (ms)	read speedup	binary size (B)	pickle size (B)	pandapower json size (B)	read pandapower file (ms, reference)	load_binary speedup vs pandapower file
case14	14	0.447	0.356	0.8x	0.268	0.202	0.8x	4173	3549	69741	298.53	1112.5x
case118	118	0.283	0.321	1.1x	0.158	0.243	1.5x	24140	25264	108962	297.09	1876.8x
case_illinois200	200	0.349	0.376	1.1x	0.216	0.341	1.6x	30240	31594	127041	288.33	1333.4x
case300	300	0.364	0.513	1.4x	0.214	0.386	1.8x	51015	53853	161723	288.39	1346.8x
case1354pegase	1354	0.411	1.844	4.5x	0.541	1.2	2.2x	230016	256682	514421	312.01	576.8x
case1888rte	1888	0.425	2.331	5.5x	0.484	1.562	3.2x	270866	303865	656133	322.06	665.0x
case2848rte	2848	0.469	3.501	7.5x	0.617	2.262	3.7x	406079	457422	951381	327.78	531.3x
case2869pegase	2869	0.637	4.021	6.3x	0.586	2.609	4.5x	507901	572650	1061979	335.8	573.2x
case3120sp	3120	0.788	3.008	3.8x	0.465	1.951	4.2x	381629	435463	961372	334.31	718.8x
case6495rte	6495	0.719	10.046	14.0x	1.199	4.618	3.9x	936004	1063398	2140446	370.48	308.9x
case6515rte	6515	0.819	9.83	12.0x	1.281	4.488	3.5x	938836	1066405	2170208	371.06	289.7x
case9241pegase	9241	1.138	15.665	13.8x	1.641	8.987	5.5x	1729197	1956152	3458084	419.07	255.4x

Two things stand out:

• The speedup of save_binary / load_binary over pickle grows with grid size: at small grids (case14) the two formats are roughly on par (pickle’s fixed per-call overhead dominates), but at case9241pegase (~9200 buses) save_binary is 14x faster to write and load_binary is 9x faster to read than pickle, and the binary file is consistently 10-15% smaller.

• Reading the original pandapower json file (pp.from_json + conversion) has a large, roughly constant overhead (~300-440 ms) essentially independent of grid size, dominated by pandapower’s own json (de)serialization rather than by the (fast) lightsim2grid conversion itself. This makes load_binary several hundred to over a thousand times faster than re-reading a pandapower file from scratch – the intended use case for this feature: keep a fast, lightsim2grid-native snapshot of an already-converted grid around, instead of re-running the full pandapower/pypowsybl conversion every time.

Note

load_binary speedups reported here are versus reading the original source format (pandapower json), not versus init_from_pandapower alone: the pandapower json reader itself is the dominant cost, not the lightsim2grid conversion. See Benchmarks (grid size) for a size scan of the raw powerflow solver speed (a different topic than serialization).