Skip to content

sliced_wasserstein

Module for sliced Wasserstein privacy losses.

The sliced Wasserstein distance is the privacy objective that trains a PrivacyEncoder. It drives every token's per-sample conditional distribution of transformed embeddings toward one shared, token-independent reference distribution. Once every token's distribution matches the same reference, the transformed embeddings no longer carry identifiable per-token clusters for an attacker to exploit, which is the privacy property being bought.

The reference can be a fixed standard normal, or, via MatchedGaussianTarget, a zero-mean Gaussian whose covariance is matched to the running covariance of the observed embeddings themselves.

Classes:

Name Description
MatchedGaussianTarget

Maintains a running estimate of the covariance of a pooled embedding stream, and samples a zero-mean Gaussian with that covariance.

Functions:

Name Description
sliced_wasserstein_distance

Compute the sliced 1-Wasserstein distance between two batches of samples.

MatchedGaussianTarget

Maintains a running estimate of the covariance of a pooled embedding stream, and samples a zero-mean Gaussian with that covariance.

The privacy objective only needs every token's conditional distribution to converge to some fixed, token-independent reference; which reference is arbitrary. A standard normal is the simplest choice, but transformer token embeddings are well documented to be anisotropic in aggregate, so forcing convergence to an isotropic ball is a strictly harder ask than "agree with each other" -- it spends encoder capacity, and utility, on a shape change that buys no extra privacy. Sampling the reference from a zero-mean Gaussian with the pooled embeddings' own covariance keeps the same indistinguishability property while asking for a shape the inputs can plausibly take on.

update folds each fresh batch into an exponential moving average of the full dim x dim covariance, rather than a plain running average, so the estimate keeps a fixed effective averaging window instead of an ever-shrinking per-step contribution as the step count grows. The first update seeds the estimate directly. Cholesky factorization is cubic in dim, so maybe_refresh only recomputes the factor sample draws from once refresh_every calls to update have elapsed since the last refresh, while the covariance estimate itself changes on every update. The first refresh is the exception: it fires as soon as an estimate exists, so that sample does not silently draw from the identity-covariance initialization.

Note

scale multiplies the Cholesky factor, equivalently multiplying the sampled target's covariance by scale ** 2. Matching the real aggregate covariance is a much easier target than a standard normal in a way that can backfire: the encoder can satisfy it by inflating each token's own local spread only slightly, which does nothing to push different tokens' outputs toward overlapping. A standard normal's concentration-of-measure-driven scale happens to be so much larger than that natural spread that reaching it floods real separation in noise as a side effect. scale recreates that flooding by demanding a proportionally larger version of the real, correctly-shaped covariance, without reintroducing the standard normal's unrelated, dimension-driven distortion of the target's absolute scale. Its right value is empirical and varies with the model and data.

Warning

Under data-parallel training this is intended to be rank-local state, computed from each rank's own disjoint slice of tokens and never synchronized: like the projection directions and target draws it feeds, it is local randomness that only ever influences that rank's own loss before gradients are all-reduced, not a model parameter. Every rank's estimate converges to approximately the same population statistics anyway, since the shards are drawn from the same token population.

Note

This is a plain Python object, so a module that holds it as an attribute does not register it: data-parallel wrappers never synchronize its running statistics, and module conversions such as .to(torch.bfloat16), .half(), or a precision plugin's never cast them. The holding module's state_dict does not include them either, so save this target's state_dict with each checkpoint, restore it with load_state_dict on resume, and move the statistics between devices with to.

Note

The bookkeeping that decides whether update seeds or folds in a batch and whether maybe_refresh is due is held in plain Python attributes, so that neither call synchronizes the host with the device unless a refresh is due.

Warning

The running statistics summarize user embeddings. They are as sensitive as any other Stained Glass Transform parameter and must not be logged or persisted outside of the training checkpoint.

Attributes:

Name Type Description
covariance Tensor

The running covariance, of shape (dim, dim), without scale applied.

cholesky_factor Tensor

The factor sample draws from: scale times the Cholesky factor of covariance as of the last refresh.

initialized bool

Whether an estimate exists, i.e. whether update has been called since construction or the last reset.

never_refreshed bool

Whether the factor has yet to be computed since construction or the last reset.

steps_since_refresh int

The number of update calls since the factor was last computed.

Parameters:

Name Type Description Default

dim

int

The dimensionality of the embeddings whose distribution to track.

required

momentum

float

The exponential moving average weight given to each fresh batch. Smaller values average over a longer window.

required

eps

float

The multiple of the identity added to the covariance before factorizing it, for numerical stability. If the factorization fails, it is retried with this multiple raised tenfold, up to three times. The value needed grows with the scale of the embeddings.

required

refresh_every

int

The number of update calls after which maybe_refresh recomputes the Cholesky factor.

required

scale

float

The factor to multiply the Cholesky factor by. See the note above.

required

device

device

The device to materialize the running statistics on. Move them later with to.

required

dtype

dtype

The dtype to accumulate the running statistics in, either float32 or float64. Accumulation is done in this dtype regardless of the dtype of the samples passed to update, and regardless of any enclosing autocast region, e.g. Lightning's bf16-mixed training step.

float32

Raises:

Type Description
ValueError

If dim is not strictly positive.

ValueError

If momentum does not lie in (0, 1].

ValueError

If eps is not positive and finite.

ValueError

If refresh_every is not strictly positive.

ValueError

If scale is not positive and finite.

ValueError

If dtype is neither float32 nor float64.

Examples:

>>> target = MatchedGaussianTarget(
...     dim=4,
...     momentum=0.01,
...     eps=1e-4,
...     refresh_every=2,
...     scale=20.0,
...     device=torch.device("cpu"),
... )
>>> generator = torch.Generator().manual_seed(0)
>>> embeddings = torch.randn(256, 4, generator=generator) * 3.0
>>> target.update(embeddings)
>>> target.maybe_refresh()
>>> samples = target.sample((8, 16, 4), generator=generator)
>>> samples.shape
torch.Size([8, 16, 4])

Save the target's state with each checkpoint, and restore it on resume:

>>> checkpoint = {"sliced_wasserstein_target": target.state_dict()}
>>> restored = MatchedGaussianTarget(
...     dim=4,
...     momentum=0.01,
...     eps=1e-4,
...     refresh_every=2,
...     scale=20.0,
...     device=torch.device("cpu"),
... )
>>> restored.load_state_dict(checkpoint["sliced_wasserstein_target"])
>>> torch.equal(restored.cholesky_factor, target.cholesky_factor)
True

Warning

This API is experimental and subject to change: The matched Gaussian target API is currently under development and is subject to change.

Added in version v4.7.0.

Methods:

Name Description
load_state_dict

Restore the running statistics and bookkeeping returned by state_dict, keeping this target's device and dtype.

maybe_refresh

Recompute the Cholesky factor that sample draws from, if it is due.

reset

Reinitialize the running statistics, discarding everything observed so far.

sample

Draw zero-mean samples whose covariance matches the current running estimate.

state_dict

Return the running statistics and the update and refresh bookkeeping, for checkpointing.

to

Move the running statistics to device, keeping their dtype.

update

Fold a fresh batch of samples into the running covariance.

load_state_dict

load_state_dict(state: _MatchedGaussianTargetState) -> None

Restore the running statistics and bookkeeping returned by state_dict, keeping this target's device and dtype.

Parameters:

Name Type Description Default

state

_MatchedGaussianTargetState

The state to restore.

required

Raises:

Type Description
ValueError

If a tensor in state does not have the shape of the corresponding statistic of this target. Nothing is restored.

maybe_refresh

maybe_refresh(force: bool = False) -> None

Recompute the Cholesky factor that sample draws from, if it is due.

The factor is recomputed if force is set, if this is the first refresh since update was ever called, or if refresh_every updates have elapsed since the last refresh. This is a no-op before the first update.

covariance itself is left holding the true, unscaled running estimate; scale is applied only to the Cholesky factor, so diagnostics that read covariance always see the real aggregate statistics rather than an inflated stand-in.

The covariance is factorized with eps times the identity added. A covariance estimated from fewer rows than dim, counting all leading dimensions of the samples, is rank-deficient, most likely on the first refresh, which factorizes the single batch that seeded the estimate. If rounding then leaves it not positive-definite, the factorization is retried with a tenfold larger multiple, up to three times, and a warning names the multiple that succeeded. Batches with well over dim rows, e.g. twice as many, give a full-rank estimate that factorizes at any embedding scale, unless the embeddings' own covariance is nearly singular.

Parameters:

Name Type Description Default

force

bool

Whether to recompute the factor regardless of how many updates have elapsed.

False

Raises:

Type Description
LinAlgError

If the covariance is not positive-definite even with the largest multiple of the identity added. The previous factor and refresh bookkeeping are left in place.

reset

reset() -> None

Reinitialize the running statistics, discarding everything observed so far.

sample

sample(
    shape: tuple[int, ...],
    device: device | None = None,
    dtype: dtype | None = None,
    generator: Generator | None = None,
) -> torch.Tensor

Draw zero-mean samples whose covariance matches the current running estimate.

Parameters:

Name Type Description Default

shape

tuple[int, ...]

The shape of the samples to draw, whose last dimension must be dim.

required

device

device | None

The device to return the samples on. If None, they are returned on the running statistics' device.

None

dtype

dtype | None

The dtype to return the samples in. If None, they are returned in the running statistics' dtype. The mapping itself is always computed in the accumulator's dtype for numerical stability, even inside an autocast region.

None

generator

Generator | None

An optional generator to draw the standard normal samples with, for reproducible losses. Must be on the running statistics' device, not device, since the samples are drawn there before being moved to device.

None

Returns:

Type Description
torch.Tensor

The sampled tensor, of the requested shape.

state_dict

state_dict() -> _MatchedGaussianTargetState

Return the running statistics and the update and refresh bookkeeping, for checkpointing.

The tensors are the target's own, not copies: update and maybe_refresh modify them in place, so serialize the state before calling either again.

Warning

The running statistics summarize user embeddings. Persist them only within the training checkpoint.

Returns:

Type Description
_MatchedGaussianTargetState

The state that load_state_dict restores.

to

to(device: device) -> Self

Move the running statistics to device, keeping their dtype.

Parameters:

Name Type Description Default

device

device

The device to move the running statistics to.

required

Returns:

Type Description
Self

This target.

update

update(samples: Tensor) -> None

Fold a fresh batch of samples into the running covariance.

Token identity is deliberately discarded: only the aggregate distribution of the pooled stream matters here. Update from the observed embeddings before the post-processor, never from its output, or the target chases the still-training post-processor's own distribution.

Samples are detached and cast to the accumulator's device and dtype, so a training batch can be passed directly, and the statistics are computed in that dtype even inside an autocast region.

Parameters:

Name Type Description Default

samples

Tensor

The observed embeddings, of shape (..., dim).

required

Raises:

Type Description
ValueError

If samples is empty.

sliced_wasserstein_distance

sliced_wasserstein_distance(
    input: Tensor,
    target: Tensor,
    num_projections: int,
    generator: Generator | None = None,
) -> torch.Tensor

Compute the sliced 1-Wasserstein distance between two batches of samples.

Both tensors are projected onto num_projections random unit directions, the projections are sorted along the samples axis, and the mean absolute difference between the sorted projections is returned. Sorting along the samples axis is what makes this a distance between distributions rather than between paired samples: the sample ordering within each distribution is irrelevant.

The leading dimensions are batch dimensions -- typically one per token, so that the returned distance averages each token's own conditional distribution against its own draw from the reference distribution.

Parameters:

Name Type Description Default

input

Tensor

The observed samples, of shape (..., num_samples, dim).

required

target

Tensor

Samples drawn from the reference distribution, of the same shape as input. Draw these with MatchedGaussianTarget.sample, or as torch.randn_like(input) for a standard normal reference.

required

num_projections

int

The number of random directions to project onto. Higher values reduce the variance of the estimate at a proportional cost.

required

generator

Generator | None

An optional generator to draw the projection directions with, for reproducible losses. Must be on input's device. If None, the global RNG state of input's device is used.

None

Returns:

Type Description
torch.Tensor

The scalar sliced 1-Wasserstein distance, in input's dtype.

Raises:

Type Description
ValueError

If num_projections is not strictly positive.

ValueError

If input has fewer than two dimensions.

ValueError

If input and target do not have the same shape.

ValueError

If input is empty.

Examples:

>>> generator = torch.Generator().manual_seed(0)
>>> samples = torch.randn(4, 32, 8, generator=generator)
>>> distance = sliced_wasserstein_distance(
...     samples, torch.randn_like(samples), num_projections=64, generator=generator
... )
>>> distance.shape
torch.Size([])

The distance between a batch of samples and itself is zero, since sorting makes the comparison order-independent:

>>> zero = sliced_wasserstein_distance(
...     samples, samples, num_projections=64, generator=generator
... )
>>> bool(zero.abs() < 1e-6)
True

Added in version v4.7.0.