Skip to content

privacy_encoder_svd

vLLM general plugin that reconstructs prompt_embeds sent as SVD coefficients, with the privacy encoder SVD at SG_PRIVACY_ENCODER_SVD_PATH.

A client may send each transformed token as its leading K SVD coefficients, in an SVDCoefficientsPayload. The server reconstructs hidden_size-dimensional embeddings from them as coefficients @ singular_vectors[:K] + mean, with the SVD of the privacy encoder's output at SG_PRIVACY_ENCODER_SVD_PATH. One SVD decodes every K up to its number of singular vectors, so the client chooses K. The reconstruction only ever recovers the projection of the encoder's output onto the subspace the client transmits. Each payload carries the ID of the SVD its coefficients were computed with, and coefficients computed with any SVD but the server's are refused: reconstructed, they would degrade the model's output without any error.

Register via the vllm.general_plugins entry point (declared in pyproject.toml), which vLLM calls in every one of its processes. The plugin checks the SVD and computes its ID, then registers a decoder for SVD coefficients payloads with the prompt-embeds loader. It decodes dense coefficients itself, and TurboQuant-compressed coefficients with the loader, so they decode wherever the TurboQuant plugin is loaded too. Coefficients in any other form, another SVD coefficients payload in particular, are refused before anything is decoded.

Only the process that decodes requests needs the SVD's tensors, and only on the device it decodes on, so no process keeps them from startup. A process reads them from the file again the first time it decodes, checks that they are still the SVD it started with, and keeps them on the decode device alone.

The file is the safetensors file Stained Glass Core's PrivacyEncoderSVD.from_file loads. Only its tensors are read, so this package does not depend on Stained Glass Core.

Errors raised while loading the SVD stop the server from starting, and name the file. Errors raised while serving a request are returned to the client, so they never name it: the operator finds it in the log.

Warning

Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, vLLM loads the plugin automatically.

Classes:

Name Description
PrivacyEncoderSVD

The SVD of the output of a client's privacy encoder, which reconstructs embeddings from their leading SVD coefficients.

Functions:

Name Description
configured_svd

Return the SVD this process decodes with, checked in SG_PRIVACY_ENCODER_SVD_PATH on first use.

register

Check the SVD at SG_PRIVACY_ENCODER_SVD_PATH, and have vLLM's prompt-embeds loader reconstruct embeddings sent as SVD coefficients.

Attributes:

Name Type Description
SG_PRIVACY_ENCODER_SVD_PATH_ENV_VAR Final[str]

Path to the SVD of the privacy encoder's output. Unset, requests carrying SVD coefficients are refused.

SG_PRIVACY_ENCODER_SVD_PATH_ENV_VAR module-attribute

SG_PRIVACY_ENCODER_SVD_PATH_ENV_VAR: Final[str] = (
    "SG_PRIVACY_ENCODER_SVD_PATH"
)

Path to the SVD of the privacy encoder's output. Unset, requests carrying SVD coefficients are refused.

PrivacyEncoderSVD dataclass

The SVD of the output of a client's privacy encoder, which reconstructs embeddings from their leading SVD coefficients.

The server-side counterpart of Stained Glass Core's PrivacyEncoderSVD, checked in the same file by from_file. It holds what identifies the SVD, not its tensors: vLLM loads this plugin in every one of its processes, and only the one that decodes requests needs them. That process reads them from the file again the first time it decodes, and keeps them on the device it decodes on alone.

Methods:

Name Description
check_coefficient_count

Check that this SVD decodes num_coefficients leading SVD coefficients per token.

decompress

Reconstruct embeddings from the leading K SVD coefficients of each token, as coefficients @ singular_vectors[:K] + mean.

from_file

Check and identify the SVD in the safetensors file Stained Glass Core's PrivacyEncoderSVD.from_file loads.

Attributes:

Name Type Description
hidden_size int

The size of the embeddings this SVD reconstructs.

path str

The file the SVD was loaded from. Only ever logged: errors returned to clients never name it.

svd_id str

The ID of the SVD, as compute_svd_id computes it.

svd_rank int

The number of singular vectors, which is the most SVD coefficients per token this SVD decodes.

hidden_size instance-attribute

hidden_size: int

The size of the embeddings this SVD reconstructs.

path instance-attribute

path: str

The file the SVD was loaded from. Only ever logged: errors returned to clients never name it.

svd_id instance-attribute

svd_id: str

The ID of the SVD, as compute_svd_id computes it.

Clients send the ID of the SVD they computed coefficients with, which must be this one. Decoding checks that the file still holds the SVD of this ID.

svd_rank instance-attribute

svd_rank: int

The number of singular vectors, which is the most SVD coefficients per token this SVD decodes.

check_coefficient_count

check_coefficient_count(num_coefficients: int) -> None

Check that this SVD decodes num_coefficients leading SVD coefficients per token.

Parameters:

Name Type Description Default

num_coefficients

int

The number of SVD coefficients per token a payload carries.

required

Raises:

Type Description
VLLMValidationError

If num_coefficients is not between 1 and svd_rank.

decompress

decompress(
    coefficients: Tensor,
    *,
    device: device,
    out_dtype: dtype,
    out_device: device,
) -> torch.Tensor

Reconstruct embeddings from the leading K SVD coefficients of each token, as coefficients @ singular_vectors[:K] + mean.

Runs on device in float32, a chunk of tokens at a time, so the memory it takes on device stays flat in the number of tokens. Each chunk is narrowed to out_dtype on device and written into the result: into a CPU result from CUDA, through pinned staging buffers, copying each chunk to the host while the next one is reconstructed. The first decode on device reads the SVD's tensors from its file onto device, where later decodes find them.

Parameters:

Name Type Description Default

coefficients

Tensor

The coefficients, of shape (num_tokens, K).

required

device

device

The device to decode on.

required

out_dtype

dtype

The dtype of the result.

required

out_device

device

The device of the result.

required

Returns:

Type Description
torch.Tensor

The reconstructed embeddings, of shape (num_tokens, hidden_size), in out_dtype on out_device.

Raises:

Type Description
VLLMValidationError

If K is not between 1 and svd_rank, the SVD's file no longer holds this SVD, or decoding runs out of memory, reading the SVD onto device and allocating the result included.

from_file classmethod

from_file(path: str) -> Self

Check and identify the SVD in the safetensors file Stained Glass Core's PrivacyEncoderSVD.from_file loads.

The file holds the mean of the privacy encoder's output, of shape (hidden_size,), and its leading singular_vectors, as orthonormal rows sorted by decreasing singular value, of shape (svd_rank, hidden_size), both in float32. Only those two tensors are read: any other tensors and the file's metadata are ignored. They are let go once the SVD is identified.

Parameters:

Name Type Description Default

path

str

Path to the SVD file.

required

Returns:

Type Description
Self

The SVD, which holds its shape and its ID, but none of its tensors.

Raises:

Type Description
ValueError

If the file cannot be loaded, does not hold the mean and singular vectors, or they do not have the expected shapes or are not float32.

configured_svd cached

configured_svd() -> PrivacyEncoderSVD | None

Return the SVD this process decodes with, checked in SG_PRIVACY_ENCODER_SVD_PATH on first use.

The plugin calls this when it registers, so a missing or malformed file stops the server from starting, and requests are checked against the ID of the SVD it held then. Only its shape and ID are kept: the first decode reads its tensors again.

Returns:

Type Description
PrivacyEncoderSVD | None

The SVD, or None when SG_PRIVACY_ENCODER_SVD_PATH is unset or empty.

Raises:

Type Description
ValueError

If the file cannot be loaded or does not hold a valid SVD.

register

register() -> None

Check the SVD at SG_PRIVACY_ENCODER_SVD_PATH, and have vLLM's prompt-embeds loader reconstruct embeddings sent as SVD coefficients.

Called automatically by vLLM's plugin system at startup in all processes. Safe to call multiple times (idempotent).

The SVD is checked (see configured_svd) before the decoder is registered, and its tensors are let go: this runs in every vLLM process, and only the one that decodes reads them again. vLLM does not catch errors raised by general plugins, so a missing or malformed file stops the server from starting, instead of failing every request that carries SVD coefficients. Without an SVD configured, the decoder is registered all the same, so that such requests are refused with the variable to set.

Raises:

Type Description
ValueError

If SG_PRIVACY_ENCODER_SVD_PATH is set to a file that cannot be loaded or does not hold a valid SVD.