sliced_wasserstein
Module for sliced Wasserstein privacy losses.
The sliced Wasserstein distance is the privacy objective that trains a
PrivacyEncoder. It drives every token's per-sample conditional
distribution of transformed embeddings toward one shared, token-independent reference distribution. Once every token's distribution
matches the same reference, the transformed embeddings no longer carry identifiable per-token clusters for an attacker to exploit, which
is the privacy property being bought.
The reference can be a fixed standard normal, or, via
MatchedGaussianTarget, a zero-mean Gaussian whose covariance is matched
to the running covariance of the observed embeddings themselves.
Classes:
| Name | Description |
|---|---|
MatchedGaussianTarget |
Maintains a running estimate of the covariance of a pooled embedding stream, and samples a zero-mean Gaussian with that covariance. |
Functions:
| Name | Description |
|---|---|
sliced_wasserstein_distance |
Compute the sliced 1-Wasserstein distance between two batches of samples. |
MatchedGaussianTarget
¶
Maintains a running estimate of the covariance of a pooled embedding stream, and samples a zero-mean Gaussian with that covariance.
The privacy objective only needs every token's conditional distribution to converge to some fixed, token-independent reference; which reference is arbitrary. A standard normal is the simplest choice, but transformer token embeddings are well documented to be anisotropic in aggregate, so forcing convergence to an isotropic ball is a strictly harder ask than "agree with each other" -- it spends encoder capacity, and utility, on a shape change that buys no extra privacy. Sampling the reference from a zero-mean Gaussian with the pooled embeddings' own covariance keeps the same indistinguishability property while asking for a shape the inputs can plausibly take on.
update folds each fresh batch into an exponential moving average of the full dim x dim covariance, rather than a plain
running average, so the estimate keeps a fixed effective averaging window instead of an ever-shrinking per-step contribution as the
step count grows. The first update seeds the estimate directly. Cholesky factorization is cubic in dim, so maybe_refresh only
recomputes the factor sample draws from once refresh_every calls to update have elapsed since the last refresh, while the
covariance estimate itself changes on every update. The first refresh is the exception: it fires as soon as an estimate exists, so
that sample does not silently draw from the identity-covariance initialization.
Note
scale multiplies the Cholesky factor, equivalently multiplying the sampled target's covariance by scale ** 2. Matching the real
aggregate covariance is a much easier target than a standard normal in a way that can backfire: the encoder can satisfy it by
inflating each token's own local spread only slightly, which does nothing to push different tokens' outputs toward overlapping. A
standard normal's concentration-of-measure-driven scale happens to be so much larger than that natural spread that reaching it
floods real separation in noise as a side effect. scale recreates that flooding by demanding a proportionally larger version of
the real, correctly-shaped covariance, without reintroducing the standard normal's unrelated, dimension-driven distortion of the
target's absolute scale. Its right value is empirical and varies with the model and data.
Warning
Under data-parallel training this is intended to be rank-local state, computed from each rank's own disjoint slice of tokens and never synchronized: like the projection directions and target draws it feeds, it is local randomness that only ever influences that rank's own loss before gradients are all-reduced, not a model parameter. Every rank's estimate converges to approximately the same population statistics anyway, since the shards are drawn from the same token population.
Note
This is a plain Python object, so a module that holds it as an attribute does not register it: data-parallel wrappers never
synchronize its running statistics, and module conversions such as .to(torch.bfloat16), .half(), or a precision plugin's never
cast them. The holding module's state_dict does not include them either, so save this target's state_dict with each checkpoint,
restore it with load_state_dict on resume, and move the statistics between devices with to.
Note
The bookkeeping that decides whether update seeds or folds in a batch and whether maybe_refresh is due is held in plain Python
attributes, so that neither call synchronizes the host with the device unless a refresh is due.
Warning
The running statistics summarize user embeddings. They are as sensitive as any other Stained Glass Transform parameter and must not be logged or persisted outside of the training checkpoint.
Attributes:
| Name | Type | Description |
|---|---|---|
covariance |
Tensor
|
The running covariance, of shape |
cholesky_factor |
Tensor
|
The factor |
initialized |
bool
|
Whether an estimate exists, i.e. whether |
never_refreshed |
bool
|
Whether the factor has yet to be computed since construction or the last |
steps_since_refresh |
int
|
The number of |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
int
|
The dimensionality of the embeddings whose distribution to track. |
required |
|
float
|
The exponential moving average weight given to each fresh batch. Smaller values average over a longer window. |
required |
|
float
|
The multiple of the identity added to the covariance before factorizing it, for numerical stability. If the factorization fails, it is retried with this multiple raised tenfold, up to three times. The value needed grows with the scale of the embeddings. |
required |
|
int
|
The number of |
required |
|
float
|
The factor to multiply the Cholesky factor by. See the note above. |
required |
|
device
|
The device to materialize the running statistics on. Move them later with |
required |
|
dtype
|
The dtype to accumulate the running statistics in, either float32 or float64. Accumulation is done in this dtype regardless
of the dtype of the samples passed to |
float32
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
Examples:
>>> target = MatchedGaussianTarget(
... dim=4,
... momentum=0.01,
... eps=1e-4,
... refresh_every=2,
... scale=20.0,
... device=torch.device("cpu"),
... )
>>> generator = torch.Generator().manual_seed(0)
>>> embeddings = torch.randn(256, 4, generator=generator) * 3.0
>>> target.update(embeddings)
>>> target.maybe_refresh()
>>> samples = target.sample((8, 16, 4), generator=generator)
>>> samples.shape
torch.Size([8, 16, 4])
Save the target's state with each checkpoint, and restore it on resume:
>>> checkpoint = {"sliced_wasserstein_target": target.state_dict()}
>>> restored = MatchedGaussianTarget(
... dim=4,
... momentum=0.01,
... eps=1e-4,
... refresh_every=2,
... scale=20.0,
... device=torch.device("cpu"),
... )
>>> restored.load_state_dict(checkpoint["sliced_wasserstein_target"])
>>> torch.equal(restored.cholesky_factor, target.cholesky_factor)
True
Warning
This API is experimental and subject to change: The matched Gaussian target API is currently under development and is subject to change.
Added in version v4.7.0.
Methods:
| Name | Description |
|---|---|
load_state_dict |
Restore the running statistics and bookkeeping returned by |
maybe_refresh |
Recompute the Cholesky factor that |
reset |
Reinitialize the running statistics, discarding everything observed so far. |
sample |
Draw zero-mean samples whose covariance matches the current running estimate. |
state_dict |
Return the running statistics and the update and refresh bookkeeping, for checkpointing. |
to |
Move the running statistics to |
update |
Fold a fresh batch of samples into the running covariance. |
load_state_dict
¶
load_state_dict(state: _MatchedGaussianTargetState) -> None
Restore the running statistics and bookkeeping returned by state_dict, keeping this target's device and dtype.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
_MatchedGaussianTargetState
|
The state to restore. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a tensor in |
maybe_refresh
¶
Recompute the Cholesky factor that sample draws from, if it is due.
The factor is recomputed if force is set, if this is the first refresh since update was ever called, or if refresh_every
updates have elapsed since the last refresh. This is a no-op before the first update.
covariance itself is left holding the true, unscaled running estimate; scale is applied only to the Cholesky factor, so
diagnostics that read covariance always see the real aggregate statistics rather than an inflated stand-in.
The covariance is factorized with eps times the identity added. A covariance estimated from fewer rows than dim, counting all
leading dimensions of the samples, is rank-deficient, most likely on the first refresh, which factorizes the single batch that
seeded the estimate. If rounding then leaves it not positive-definite, the factorization is retried with a tenfold larger multiple,
up to three times, and a warning names the multiple that succeeded. Batches with well over dim rows, e.g. twice as many, give a
full-rank estimate that factorizes at any embedding scale, unless the embeddings' own covariance is nearly singular.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
bool
|
Whether to recompute the factor regardless of how many updates have elapsed. |
False
|
Raises:
| Type | Description |
|---|---|
LinAlgError
|
If the covariance is not positive-definite even with the largest multiple of the identity added. The previous factor and refresh bookkeeping are left in place. |
sample
¶
sample(
shape: tuple[int, ...],
device: device | None = None,
dtype: dtype | None = None,
generator: Generator | None = None,
) -> torch.Tensor
Draw zero-mean samples whose covariance matches the current running estimate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
tuple[int, ...]
|
The shape of the samples to draw, whose last dimension must be |
required |
|
device | None
|
The device to return the samples on. If |
None
|
|
dtype | None
|
The dtype to return the samples in. If |
None
|
|
Generator | None
|
An optional generator to draw the standard normal samples with, for reproducible losses. Must be on the running
statistics' device, not |
None
|
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The sampled tensor, of the requested |
state_dict
¶
Return the running statistics and the update and refresh bookkeeping, for checkpointing.
The tensors are the target's own, not copies: update and maybe_refresh modify them in place, so serialize the state before
calling either again.
Warning
The running statistics summarize user embeddings. Persist them only within the training checkpoint.
Returns:
| Type | Description |
|---|---|
_MatchedGaussianTargetState
|
The state that |
to
¶
update
¶
Fold a fresh batch of samples into the running covariance.
Token identity is deliberately discarded: only the aggregate distribution of the pooled stream matters here. Update from the observed embeddings before the post-processor, never from its output, or the target chases the still-training post-processor's own distribution.
Samples are detached and cast to the accumulator's device and dtype, so a training batch can be passed directly, and the statistics are computed in that dtype even inside an autocast region.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The observed embeddings, of shape |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
sliced_wasserstein_distance
¶
sliced_wasserstein_distance(
input: Tensor,
target: Tensor,
num_projections: int,
generator: Generator | None = None,
) -> torch.Tensor
Compute the sliced 1-Wasserstein distance between two batches of samples.
Both tensors are projected onto num_projections random unit directions, the projections are sorted along the samples axis, and the
mean absolute difference between the sorted projections is returned. Sorting along the samples axis is what makes this a distance
between distributions rather than between paired samples: the sample ordering within each distribution is irrelevant.
The leading dimensions are batch dimensions -- typically one per token, so that the returned distance averages each token's own conditional distribution against its own draw from the reference distribution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The observed samples, of shape |
required |
|
Tensor
|
Samples drawn from the reference distribution, of the same shape as |
required |
|
int
|
The number of random directions to project onto. Higher values reduce the variance of the estimate at a proportional cost. |
required |
|
Generator | None
|
An optional generator to draw the projection directions with, for reproducible losses. Must be on |
None
|
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The scalar sliced 1-Wasserstein distance, in |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
Examples:
>>> generator = torch.Generator().manual_seed(0)
>>> samples = torch.randn(4, 32, 8, generator=generator)
>>> distance = sliced_wasserstein_distance(
... samples, torch.randn_like(samples), num_projections=64, generator=generator
... )
>>> distance.shape
torch.Size([])
The distance between a batch of samples and itself is zero, since sorting makes the comparison order-independent:
>>> zero = sliced_wasserstein_distance(
... samples, samples, num_projections=64, generator=generator
... )
>>> bool(zero.abs() < 1e-6)
True
Added in version v4.7.0.