privacy_encoder_svd
vLLM general plugin that reconstructs prompt_embeds sent as SVD coefficients, with the privacy encoder SVD at SG_PRIVACY_ENCODER_SVD_PATH.
A client may send each transformed token as its leading K SVD coefficients, in an
SVDCoefficientsPayload. The server reconstructs
hidden_size-dimensional embeddings from them as coefficients @ singular_vectors[:K] + mean, with the SVD of the privacy encoder's
output at SG_PRIVACY_ENCODER_SVD_PATH. One SVD decodes every K up to its number of singular vectors, so the client chooses K. The
reconstruction only ever recovers the projection of the encoder's output onto the subspace the client transmits. Each payload carries the
ID of the SVD its coefficients were computed with, and coefficients computed with any SVD but the server's are refused: reconstructed, they
would degrade the model's output without any error.
Register via the vllm.general_plugins entry point (declared in pyproject.toml), which vLLM calls in every one of its processes. The plugin
checks the SVD and computes its ID, then registers a decoder for SVD coefficients payloads with the
prompt-embeds loader. It decodes dense coefficients itself, and
TurboQuant-compressed coefficients with the loader, so they decode wherever the TurboQuant plugin is loaded too. Coefficients in any other
form, another SVD coefficients payload in particular, are refused before anything is decoded.
Only the process that decodes requests needs the SVD's tensors, and only on the device it decodes on, so no process keeps them from startup. A process reads them from the file again the first time it decodes, checks that they are still the SVD it started with, and keeps them on the decode device alone.
The file is the safetensors file Stained Glass Core's PrivacyEncoderSVD.from_file loads. Only its tensors are read, so this package does
not depend on Stained Glass Core.
Errors raised while loading the SVD stop the server from starting, and name the file. Errors raised while serving a request are returned to the client, so they never name it: the operator finds it in the log.
Warning
Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, vLLM
loads the plugin automatically.
Classes:
| Name | Description |
|---|---|
PrivacyEncoderSVD |
The SVD of the output of a client's privacy encoder, which reconstructs embeddings from their leading SVD coefficients. |
Functions:
| Name | Description |
|---|---|
configured_svd |
Return the SVD this process decodes with, checked in |
register |
Check the SVD at |
Attributes:
| Name | Type | Description |
|---|---|---|
SG_PRIVACY_ENCODER_SVD_PATH_ENV_VAR |
Final[str]
|
Path to the SVD of the privacy encoder's output. Unset, requests carrying SVD coefficients are refused. |
SG_PRIVACY_ENCODER_SVD_PATH_ENV_VAR
module-attribute
¶
Path to the SVD of the privacy encoder's output. Unset, requests carrying SVD coefficients are refused.
PrivacyEncoderSVD
dataclass
¶
The SVD of the output of a client's privacy encoder, which reconstructs embeddings from their leading SVD coefficients.
The server-side counterpart of Stained Glass Core's PrivacyEncoderSVD, checked in the same file by from_file. It holds what
identifies the SVD, not its tensors: vLLM loads this plugin in every one of its processes, and only the one that decodes requests needs
them. That process reads them from the file again the first time it decodes, and keeps them on the device it decodes on alone.
Methods:
| Name | Description |
|---|---|
check_coefficient_count |
Check that this SVD decodes |
decompress |
Reconstruct embeddings from the leading |
from_file |
Check and identify the SVD in the safetensors file Stained Glass Core's |
Attributes:
| Name | Type | Description |
|---|---|---|
hidden_size |
int
|
The size of the embeddings this SVD reconstructs. |
path |
str
|
The file the SVD was loaded from. Only ever logged: errors returned to clients never name it. |
svd_id |
str
|
The ID of the SVD, as |
svd_rank |
int
|
The number of singular vectors, which is the most SVD coefficients per token this SVD decodes. |
path
instance-attribute
¶
path: str
The file the SVD was loaded from. Only ever logged: errors returned to clients never name it.
svd_id
instance-attribute
¶
svd_id: str
The ID of the SVD, as compute_svd_id computes it.
Clients send the ID of the SVD they computed coefficients with, which must be this one. Decoding checks that the file still holds the SVD of this ID.
svd_rank
instance-attribute
¶
svd_rank: int
The number of singular vectors, which is the most SVD coefficients per token this SVD decodes.
check_coefficient_count
¶
check_coefficient_count(num_coefficients: int) -> None
Check that this SVD decodes num_coefficients leading SVD coefficients per token.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
int
|
The number of SVD coefficients per token a payload carries. |
required |
Raises:
| Type | Description |
|---|---|
VLLMValidationError
|
If |
decompress
¶
decompress(
coefficients: Tensor,
*,
device: device,
out_dtype: dtype,
out_device: device,
) -> torch.Tensor
Reconstruct embeddings from the leading K SVD coefficients of each token, as coefficients @ singular_vectors[:K] + mean.
Runs on device in float32, a chunk of tokens at a time, so the memory it takes on device stays flat in the number of tokens.
Each chunk is narrowed to out_dtype on device and written into the result: into a CPU result from CUDA, through pinned staging
buffers, copying each chunk to the host while the next one is reconstructed. The first decode on device reads the SVD's tensors
from its file onto device, where later decodes find them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The coefficients, of shape |
required |
|
device
|
The device to decode on. |
required |
|
dtype
|
The dtype of the result. |
required |
|
device
|
The device of the result. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The reconstructed embeddings, of shape |
Raises:
| Type | Description |
|---|---|
VLLMValidationError
|
If |
from_file
classmethod
¶
Check and identify the SVD in the safetensors file Stained Glass Core's PrivacyEncoderSVD.from_file loads.
The file holds the mean of the privacy encoder's output, of shape (hidden_size,), and its leading singular_vectors, as
orthonormal rows sorted by decreasing singular value, of shape (svd_rank, hidden_size), both in float32. Only those two tensors
are read: any other tensors and the file's metadata are ignored. They are let go once the SVD is identified.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str
|
Path to the SVD file. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The SVD, which holds its shape and its ID, but none of its tensors. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file cannot be loaded, does not hold the mean and singular vectors, or they do not have the expected shapes or are not float32. |
configured_svd
cached
¶
Return the SVD this process decodes with, checked in SG_PRIVACY_ENCODER_SVD_PATH on first use.
The plugin calls this when it registers, so a missing or malformed file stops the server from starting, and requests are checked against the ID of the SVD it held then. Only its shape and ID are kept: the first decode reads its tensors again.
Returns:
| Type | Description |
|---|---|
PrivacyEncoderSVD | None
|
The SVD, or |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file cannot be loaded or does not hold a valid SVD. |
register
¶
Check the SVD at SG_PRIVACY_ENCODER_SVD_PATH, and have vLLM's prompt-embeds loader reconstruct embeddings sent as SVD coefficients.
Called automatically by vLLM's plugin system at startup in all processes. Safe to call multiple times (idempotent).
The SVD is checked (see configured_svd) before the decoder
is registered, and its tensors are let go: this runs in every vLLM process, and only the one that decodes reads them again. vLLM does
not catch errors raised by general plugins, so a missing or malformed file stops the server from starting, instead of failing every
request that carries SVD coefficients. Without an SVD configured, the decoder is registered all the same, so that such requests are
refused with the variable to set.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |