privacy_encoder_svd
Inference-time compression of a trained privacy encoder's output to its leading SVD coefficients.
Classes:
| Name | Description |
|---|---|
PrivacyEncoderSVD |
The SVD of a trained privacy encoder's output, which compresses that output to its leading SVD coefficients at inference. |
PrivacyEncoderSVD
¶
Bases: Module
The SVD of a trained privacy encoder's output, which compresses that output to its leading SVD coefficients at inference.
The SVD is fitted after training, on the output of a trained
PrivacyEncoder over a representative dataset, and loaded at inference
with from_file. It consists of the mean of that output and its leading singular vectors: the leading left singular vectors of its
covariance, as orthonormal rows sorted by decreasing singular value.
The client compresses each embedding the noise layer transformed to its leading K coefficients in that basis with compress, for any
K up to svd_rank, and transmits only those. The model side reconstructs the embedding from them with decompress, as the
projection of the embedding onto the affine subspace spanned by the leading K singular vectors. The components of the embedding
outside of that subspace never leave the client.
The SVD is held in float32, and compress and decompress compute in float32, even inside an autocast region.
Examples:
Transmit the first two of four channels, then only the first:
>>> svd = PrivacyEncoderSVD(mean=torch.zeros(4), singular_vectors=torch.eye(4)[:2])
>>> embeddings = torch.tensor([[1.0, 2.0, 3.0, 4.0]])
>>> coefficients = svd.compress(embeddings)
>>> coefficients
tensor([[1., 2.]])
>>> svd.decompress(coefficients)
tensor([[1., 2., 0., 0.]])
>>> svd.decompress(svd.compress(embeddings, svd_rank=1))
tensor([[1., 0., 0., 0.]])
Added in version v5.2.0.
Methods:
| Name | Description |
|---|---|
__init__ |
Hold the SVD of a privacy encoder's output. |
compress |
Compress |
decompress |
Reconstruct |
from_file |
Load the SVD of a privacy encoder's output from a safetensors file. |
Attributes:
| Name | Type | Description |
|---|---|---|
dim |
int
|
The dimensionality of the privacy encoder's output. |
svd_rank |
int
|
The number of singular vectors, i.e. the largest number of SVD coefficients an embedding can be compressed to. |
svd_rank
property
¶
svd_rank: int
The number of singular vectors, i.e. the largest number of SVD coefficients an embedding can be compressed to.
__init__
¶
__init__(mean: Tensor, singular_vectors: Tensor) -> None
Hold the SVD of a privacy encoder's output.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The mean of the privacy encoder's output, in float32, of shape |
required |
|
Tensor
|
The leading singular vectors of the privacy encoder's output, in float32, as orthonormal rows sorted by
decreasing singular value, of shape |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
ValueError
|
If the rows of |
compress
¶
Compress input to its leading svd_rank SVD coefficients, (input - mean) @ singular_vectors[:svd_rank].mT.
The coefficients are computed in float32, even inside an autocast region, then cast to the dtype of input.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The privacy encoder's output, of shape |
required |
|
int | None
|
The number of leading coefficients to compress to, between 1 and |
None
|
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The coefficients, of shape |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the last dimension of |
ValueError
|
If |
decompress
¶
decompress(coefficients: Tensor) -> torch.Tensor
Reconstruct dim-dimensional embeddings from their leading SVD coefficients, coefficients @ singular_vectors[:K] + mean.
K is the number of coefficients, i.e. the size of the last dimension of coefficients. The reconstruction is the projection of
the compressed embedding onto the affine subspace spanned by the leading K singular vectors. It is computed in float32, even
inside an autocast region, then cast to the dtype of coefficients.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The leading SVD coefficients, as returned by |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The reconstructed embeddings, of shape |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
from_file
classmethod
¶
Load the SVD of a privacy encoder's output from a safetensors file.
The file holds the mean under the mean key and the singular vectors under the singular_vectors key, both in float32 as described
in __init__. Only those two tensors are read: any other tensors and the file's metadata are ignored.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str | PathLike[str]
|
The path to the file. |
required |
|
device | str
|
The device to load the SVD onto. |
'cpu'
|
Returns:
| Type | Description |
|---|---|
Self
|
The loaded SVD. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file is not a safetensors file. |
ValueError
|
If the file does not hold the |