steppingpack

steppingpack is a space efficient wrapper around MessagePack.

Instead of:

Pydantic → JSON

Think:

steppingpack → MessagePack


Space is saved by not storing the keys of values, so given:

class Foo(st.Data):
    bar: int
    qux: str

foo = Foo(bar=1, qux="WOO")

instead of storing:

{"bar": 1, "qux": "WOO"}

we store:

[1, "WOO"]

Usage

Given classes like:

from stepping import steppingpack

class DataA(steppingpack.Data):
    a: str

class DataB(steppingpack.Data):
    x: int
    many: tuple[DataA, ...]

serialize:

dumped: bytes = steppingpack.dump(DataB(...))

and deserialize:

loaded: DataB  = steppingpack.load(DataB, dumped)

Supported types

steppingpack can (de)serialize

Atom = str | int | float | bool | None | date | datetime | UUID | Enum
Value = (
    Atom |
    tuple[Value, ...] |
    frozenset[Value] |
    Dumpable |
    Data
)

Where Dumpable is any class implementing:

st_arity: ClassVar[Arity]
st_astuple: tuple[Value, ...]

It also has support for:

  • Recursive type definitions.
  • Union types. Unions of Data must have a string .st_discriminant field to tell them apart, for example:
class DataC(steppingpack.Data):
    st_discriminant: str = "DataC"
    ...

Future

  • More types will be added, notably missing at the moment is datetime.time. Some kind of FrozenDict support would be nice too, especially if it played well with other types in stepping.datatypes (think hard about maintaining key order here and its applicability to serializing ZSetPythons too).
  • Migrating between types in production will be added as part of stepping manager.
  • A version of isjsonschemasubset will be implemented to help with the above.
  • Do more Rust-ification, especially for dump_python – under the hood, steppingpack uses ormsgpack, potentially even try upstream into that.
  • Move into its own separate project?
← API
Performance →