Skip to content

privacy_encoder_svd

Inference-time compression of a trained privacy encoder's output to its leading SVD coefficients.

Classes:

Name Description
PrivacyEncoderSVD

The SVD of a trained privacy encoder's output, which compresses that output to its leading SVD coefficients at inference.

PrivacyEncoderSVD

Bases: Module

The SVD of a trained privacy encoder's output, which compresses that output to its leading SVD coefficients at inference.

The SVD is fitted after training, on the output of a trained PrivacyEncoder over a representative dataset, and loaded at inference with from_file. It consists of the mean of that output and its leading singular vectors: the leading left singular vectors of its covariance, as orthonormal rows sorted by decreasing singular value.

The client compresses each embedding the noise layer transformed to its leading K coefficients in that basis with compress, for any K up to svd_rank, and transmits only those. The model side reconstructs the embedding from them with decompress, as the projection of the embedding onto the affine subspace spanned by the leading K singular vectors. The components of the embedding outside of that subspace never leave the client.

The SVD is held in float32, and compress and decompress compute in float32, even inside an autocast region.

Examples:

Transmit the first two of four channels, then only the first:

>>> svd = PrivacyEncoderSVD(mean=torch.zeros(4), singular_vectors=torch.eye(4)[:2])
>>> embeddings = torch.tensor([[1.0, 2.0, 3.0, 4.0]])
>>> coefficients = svd.compress(embeddings)
>>> coefficients
tensor([[1., 2.]])
>>> svd.decompress(coefficients)
tensor([[1., 2., 0., 0.]])
>>> svd.decompress(svd.compress(embeddings, svd_rank=1))
tensor([[1., 0., 0., 0.]])

Added in version v5.2.0.

Methods:

Name Description
__init__

Hold the SVD of a privacy encoder's output.

compress

Compress input to its leading svd_rank SVD coefficients, (input - mean) @ singular_vectors[:svd_rank].mT.

decompress

Reconstruct dim-dimensional embeddings from their leading SVD coefficients, coefficients @ singular_vectors[:K] + mean.

from_file

Load the SVD of a privacy encoder's output from a safetensors file.

Attributes:

Name Type Description
dim int

The dimensionality of the privacy encoder's output.

svd_rank int

The number of singular vectors, i.e. the largest number of SVD coefficients an embedding can be compressed to.

dim property

dim: int

The dimensionality of the privacy encoder's output.

svd_rank property

svd_rank: int

The number of singular vectors, i.e. the largest number of SVD coefficients an embedding can be compressed to.

__init__

__init__(mean: Tensor, singular_vectors: Tensor) -> None

Hold the SVD of a privacy encoder's output.

Parameters:

Name Type Description Default

mean

Tensor

The mean of the privacy encoder's output, in float32, of shape (dim,).

required

singular_vectors

Tensor

The leading singular vectors of the privacy encoder's output, in float32, as orthonormal rows sorted by decreasing singular value, of shape (svd_rank, dim) for some 1 <= svd_rank <= dim.

required

Raises:

Type Description
ValueError

If mean or singular_vectors has the wrong shape.

ValueError

If mean or singular_vectors is not float32.

ValueError

If the rows of singular_vectors are not orthonormal.

compress

compress(
    input: Tensor, svd_rank: int | None = None
) -> torch.Tensor

Compress input to its leading svd_rank SVD coefficients, (input - mean) @ singular_vectors[:svd_rank].mT.

The coefficients are computed in float32, even inside an autocast region, then cast to the dtype of input.

Parameters:

Name Type Description Default

input

Tensor

The privacy encoder's output, of shape (..., dim).

required

svd_rank

int | None

The number of leading coefficients to compress to, between 1 and self.svd_rank inclusive. If None, all of them.

None

Returns:

Type Description
torch.Tensor

The coefficients, of shape (..., svd_rank) and of the dtype of input.

Raises:

Type Description
ValueError

If the last dimension of input is not of size dim.

ValueError

If svd_rank is not between 1 and self.svd_rank inclusive.

decompress

decompress(coefficients: Tensor) -> torch.Tensor

Reconstruct dim-dimensional embeddings from their leading SVD coefficients, coefficients @ singular_vectors[:K] + mean.

K is the number of coefficients, i.e. the size of the last dimension of coefficients. The reconstruction is the projection of the compressed embedding onto the affine subspace spanned by the leading K singular vectors. It is computed in float32, even inside an autocast region, then cast to the dtype of coefficients.

Parameters:

Name Type Description Default

coefficients

Tensor

The leading SVD coefficients, as returned by compress, of shape (..., K).

required

Returns:

Type Description
torch.Tensor

The reconstructed embeddings, of shape (..., dim) and of the dtype of coefficients.

Raises:

Type Description
ValueError

If K is not between 1 and self.svd_rank inclusive.

from_file classmethod

from_file(
    path: str | PathLike[str], device: device | str = "cpu"
) -> Self

Load the SVD of a privacy encoder's output from a safetensors file.

The file holds the mean under the mean key and the singular vectors under the singular_vectors key, both in float32 as described in __init__. Only those two tensors are read: any other tensors and the file's metadata are ignored.

Parameters:

Name Type Description Default

path

str | PathLike[str]

The path to the file.

required

device

device | str

The device to load the SVD onto.

'cpu'

Returns:

Type Description
Self

The loaded SVD.

Raises:

Type Description
ValueError

If the file is not a safetensors file.

ValueError

If the file does not hold the mean and singular_vectors tensors.