Skip to content

serialization

Utilities for serializing and deserializing large, nested dictionaries.

Classes:

Name Description
IndexFileMalformedError

Exception raised when the index file is malformed or missing required keys.

MissingIndexFileError

Exception raised when the index file is missing in the ZIP archive.

SchemaZIPSerializer

Serialize and deserialize a large, nested dictionary into a ZIP archive

Functions:

Name Description
compress_files_in_parallel

Compress multiple files' bytes with zlib, one worker thread per file.

decompress_files_in_parallel

Decompress multiple files' zlib-compressed bytes, one worker thread per file.

filter_keys_from_dict_missing_in_signature

Filter keys from a dictionary that are not in the signature of a given function.

find_non_jsonable

Recursively find non-JSON-serializable objects in a data structure.

get_fully_qualified_class_name_for_import

Get the fully qualified name of a class, including its module and any containing classes.

import_class_from_fully_qualified_name

Dynamically imports and returns a class or attribute from a fully qualified name.

resolve_cpu_worker_budget

Get the upper bound on CPU-bound worker threads for parallel shard/file processing.

IndexFileMalformedError

Bases: KeyError

Exception raised when the index file is malformed or missing required keys.

MissingIndexFileError

Bases: KeyError

Exception raised when the index file is missing in the ZIP archive.

SchemaZIPSerializer

Serialize and deserialize a large, nested dictionary into a ZIP archive (returned as bytes) based on a user-provided schema mapping of key paths to file templates. Supports dynamic per-key files via {key} in templates. Schema mapping keys are tuples representing the dictionary path, e.g. ('config', 'noise_tokenizer') Values are relative paths within the ZIP, which may include a {key} placeholder for dynamic subfiles. The ZIP will contain: - A root schema file (default: 'index.json') holding the mapping and a skeleton of the data with $ref placeholders. - One or more JSON files per the mapping. - Any additional files passed via extra_files Use dumps(data, extra_files=...) -> bytes to produce a ZIP, and loads(zip_bytes) -> (dict, extra_files) to reconstruct. Unmapped keys are included inline in the schema and preserved on deserialization.

Examples:

>>> data = {
...     "config": {
...         "settings": {"x": 1, "y": 2},
...         "noise_tokenizer": {"t1": {"a": 0.1}, "t2": {"a": 0.2}},
...     },
...     "users": {"alice": {"id": 1}, "bob": {"id": 2}},
...     "notes": {"misc": "inline"},
... }

Mappings must contain the root index file, and can contain any chain of nested keys. If a filename in the mapping contains a {key} placeholder, the serializer will create a separate file for each key in the dictionary at that path. (It is an error to use {key} in a mapping that does not point to a dictionary.) Mappings also do not need to be complete; any keys not in the mapping will be included inline in the skeleton in the index file.

>>> mapping = {
...     (): "index.json",
...     ("config", "settings"): "config/settings.json",
...     ("config", "noise_tokenizer"): "noise/{key}.json",
...     ("users",): "users.json",
... }
>>> extra_files = {"README.txt": "This is a test ZIP."}
>>> serializer = SchemaZIPSerializer(mapping, zipfile.ZIP_DEFLATED)
>>> zip_bytes = serializer.dumps(data, extra_files=extra_files)
>>> isinstance(zip_bytes, bytes)
True
>>> restored, extras = SchemaZIPSerializer.loads(zip_bytes)
>>> restored == data
True
>>> extras["README.txt"].decode() == "This is a test ZIP."
True

Inspecting ZIP contents:

>>> zf = zipfile.ZipFile(io.BytesIO(zip_bytes))
>>> sorted(zf.namelist())
['README.txt', 'config/settings.json', 'index.json', 'noise/t1.json', 'noise/t2.json', 'users.json']

Deserialization still works regardless of the mapping used:

>>> mapping2 = {
...     (): "index.json",
... }
>>> zip_bytes_2 = SchemaZIPSerializer(
...     mapping2, compression=zipfile.ZIP_DEFLATED
... ).dumps(data)
>>> data2, _ = SchemaZIPSerializer.loads(zip_bytes_2)
>>> data2 == data
True

Methods:

Name Description
__init__

Initialize the serializer with a mapping of paths to filename templates.

dumps

Serialize the data into a ZIP archive.

loads

Deserialize the ZIP archive back into its original data dictionary and extra files.

mark_extra_files_uncompressed

Return a copy of a schema file's bytes with the extra_files_zlib_compressed flag cleared.

Attributes:

Name Type Description
REF_KEY Final[str]

Key used to indicate a reference to a file in the ZIP archive. This generally should not be changed.

SCHEMA_FILENAME Final[str]

Default filename for the root schema file in the ZIP. This generally should not be changed.

REF_KEY class-attribute instance-attribute

REF_KEY: Final[str] = '$ref'

Key used to indicate a reference to a file in the ZIP archive. This generally should not be changed.

SCHEMA_FILENAME class-attribute instance-attribute

SCHEMA_FILENAME: Final[str] = 'index.json'

Default filename for the root schema file in the ZIP. This generally should not be changed.

__init__

__init__(
    mapping: SerializationSchemaMapping,
    compression: int,
    extra_files_compresslevel: int | None = None,
    extra_files_compression: int | None = None,
) -> None

Initialize the serializer with a mapping of paths to filename templates.

Parameters:

Name Type Description Default

mapping

SerializationSchemaMapping

A dictionary mapping tuples of strings (representing paths in the nested dictionary) to filename templates. The templates can include a {key} placeholder for dynamic subfiles, where each key in the subdictionary will get its own file, with the key replacing {key} in the filename.

required

compression

int

The compression method to use for the ZIP file. Usually represented by zipfile.ZIP_DEFLATED or zipfile.ZIP_STORED.

required

extra_files_compresslevel

int | None

The zlib compresslevel used to compress extra_files, when extra_files_compression requests compression. Ignored when extra_files_compression is zipfile.ZIP_STORED. Defaults to None (zlib's default level, zlib.Z_DEFAULT_COMPRESSION).

None

extra_files_compression

int | None

Whether to compress extra_files in dumps, independent of compression. Defaults to compression when None. Pass zipfile.ZIP_STORED to store extra_files uncompressed (e.g. because the caller already has pre-compressed bytes to store as-is); any other value compresses them with zlib, in parallel across one worker thread per file, before storing them -- so dumps's caller never needs to compress extra_files itself or know how many workers to use. loads reverses this automatically and transparently.

None

dumps

dumps(
    data: dict[str, Any],
    extra_files: (
        Mapping[str, str | bytes | memoryview] | None
    ) = None,
) -> bytes

Serialize the data into a ZIP archive.

Parameters:

Name Type Description Default

data

dict[str, Any]

The data to serialize. Must be a dictionary.

required

extra_files

Mapping[str, str | bytes | memoryview] | None

Optional additional files to include in the ZIP. Keys are filenames, values are file contents (str, bytes, or a zero-copy memoryview -- e.g. a slice of a larger buffer the caller doesn't want duplicated by slicing it into a new bytes object). If str, it will be encoded to bytes using UTF-8. If None, no extra files are added. Compressed (in parallel, one worker thread per file) according to extra_files_compression/extra_files_compresslevel rather than compression.

None

Returns:

Type Description
bytes

The serialized ZIP archive as bytes.

loads classmethod

loads(
    zip_bytes: bytes, index_file_name: str | None = None
) -> tuple[dict[str, Any], dict[str, bytes]]

Deserialize the ZIP archive back into its original data dictionary and extra files.

Parameters:

Name Type Description Default

zip_bytes

bytes

The ZIP archive as bytes.

required

index_file_name

str | None

The name of the root schema file in the ZIP. When not specified, defaults to the class constant. This should generally not be specified, but is exposed to allow for the ability to open old ZIP files if the class constant is changed.

None

Returns:

Type Description
tuple[dict[str, Any], dict[str, bytes]]

A tuple containing: - The reconstructed data dictionary. - A dictionary of extra files, where keys are filenames and values are file contents as bytes.

Raises:

Type Description
MissingIndexFileError

If the specified index file is missing from the ZIP archive.

IndexFileMalformedError

If the index file is malformed or missing required keys.

mark_extra_files_uncompressed classmethod

mark_extra_files_uncompressed(schema_bytes: bytes) -> bytes

Return a copy of a schema file's bytes with the extra_files_zlib_compressed flag cleared.

For a caller that decompresses a serialized archive's extra_files in place after the fact -- e.g. to hand raw file contents to a system that doesn't know about this class's zlib pre-compression, such as the Hugging Face Hub -- and then re-persists the schema file alongside them, this keeps the schema honest about what it now travels with, so a later loads on that changed archive doesn't try to reverse a compression that's no longer there.

Parameters:

Name Type Description Default

schema_bytes

bytes

The raw bytes of a schema file (i.e. SCHEMA_FILENAME), as produced by dumps.

required

Returns:

Type Description
bytes

The schema bytes, re-encoded with extra_files_zlib_compressed set to False.

compress_files_in_parallel

compress_files_in_parallel(
    files: Mapping[str, bytes | memoryview], level: int
) -> dict[str, bytes]

Compress multiple files' bytes with zlib, one worker thread per file.

Parameters:

Name Type Description Default

files

Mapping[str, bytes | memoryview]

A mapping of filename to raw bytes (or a zero-copy memoryview of a larger buffer) to compress.

required

level

int

The zlib compresslevel to use for every file.

required

Returns:

Type Description
dict[str, bytes]

A mapping of filename to zlib-compressed bytes, with the same keys as files.

decompress_files_in_parallel

decompress_files_in_parallel(
    files: Mapping[str, bytes],
) -> dict[str, bytes]

Decompress multiple files' zlib-compressed bytes, one worker thread per file.

Concurrency is capped at resolve_cpu_worker_budget() regardless of how many files are passed in, since -- unlike the writer, which controls how many files it produces -- this may be reading an arbitrary, untrusted archive with an unbounded number of entries.

Parameters:

Name Type Description Default

files

Mapping[str, bytes]

A mapping of filename to zlib-compressed bytes.

required

Returns:

Type Description
dict[str, bytes]

A mapping of filename to decompressed bytes, with the same keys as files.

filter_keys_from_dict_missing_in_signature

filter_keys_from_dict_missing_in_signature(
    state: Mapping[str, Any], func: Callable[..., Any]
) -> dict[str, Any]

Filter keys from a dictionary that are not in the signature of a given function.

This is useful for cleaning up state dictionaries before passing them to a function that may not expect all the keys.

If func accepts a **kwargs parameter (VAR_KEYWORD), all keys are considered valid and a shallow copy of state is returned without any filtering or warnings.

Parameters:

Name Type Description Default

state

Mapping[str, Any]

The original dictionary containing the state.

required

func

Callable[..., Any]

The function whose signature will be used to filter the keys.

required

Returns:

Type Description
dict[str, Any]

A new dictionary containing only the keys that are present in the function's signature, or a shallow copy of

dict[str, Any]

all keys when the function accepts **kwargs.

find_non_jsonable

find_non_jsonable(
    obj: Any,
    *,
    path: str = "$",
    seen: set[int] | None = None
) -> list[str]

Recursively find non-JSON-serializable objects in a data structure.

Parameters:

Name Type Description Default

obj

Any

The object to check for JSON serializability.

required

path

str

The JSON path to the current object, used for error reporting. Generally, this should be left as the default when calling the function, as it is used internally for recursion.

'$'

seen

set[int] | None

A set of object IDs that have already been visited to avoid infinite recursion. Generally, this should be None when calling the function, as it is used internally for recursion.

None

Returns:

Type Description
list[str]

A list of JSON paths to non-JSON-serializable objects.

get_fully_qualified_class_name_for_import

get_fully_qualified_class_name_for_import(
    cls: type[Any],
) -> str

Get the fully qualified name of a class, including its module and any containing classes.

Parameters:

Name Type Description Default

cls

type[Any]

The class to get the fully qualified name for.

required

Examples:

>>> type_str = get_fully_qualified_class_name_for_import(SchemaZIPSerializer)
>>> print(type_str)
stainedglass_core.utils.serialization.SchemaZIPSerializer
>>> imported_type = import_class_from_fully_qualified_name(type_str)
>>> print(imported_type)
<class 'stainedglass_core.utils.serialization.SchemaZIPSerializer'>

Returns:

Type Description
str

The fully qualified name of the class, in the format 'module.submodule.ClassName' or 'module.submodule.OuterClass.InnerClass'.

import_class_from_fully_qualified_name

import_class_from_fully_qualified_name(
    fully_qualified_class_name: str,
) -> Any

Dynamically imports and returns a class or attribute from a fully qualified name.

Parameters:

Name Type Description Default

fully_qualified_class_name

str

The fully qualified name of the class or attribute to import, in the format 'module.submodule.ClassName' or 'module.submodule.ClassName.attribute'.

required

Returns:

Name Type Description
Any Any

The imported class or attribute.

Raises:

Type Description
ValueError

If the provided fully qualified name does not contain a module path.

resolve_cpu_worker_budget

resolve_cpu_worker_budget() -> int

Get the upper bound on CPU-bound worker threads for parallel shard/file processing.

Defaults to os.cpu_count(), which does not reflect container CPU quotas (cgroup limits) and can over-provision threads relative to the actual allotted CPU -- or simply over-provision on a high-logical-core-count machine, adding scheduling overhead instead of speedup. Set the SGC_MAX_PARALLEL_WORKERS environment variable to cap it explicitly.

Returns:

Type Description
int

The CPU worker budget, before it is further capped by the number of items actually being

int

processed (e.g. via min(len(items), resolve_cpu_worker_budget())).

Raises:

Type Description
ValueError

If SGC_MAX_PARALLEL_WORKERS is set but is not a positive integer.