local_similarity
Module for spatially-localized structural-similarity privacy statistics.
multi_attacker_privacy_loss reduces every attacker view to one
scalar per image, so it cannot represent "most of this image is destroyed but this region is not". A region that survives un-obfuscated
moves a whole-image average by O(area_fraction) and is invisible in practice. This module keeps the spatial structure: it scores every
window of the image independently and then reduces with a tail (see
tail_mean), so a small number of surviving windows sets the value.
Design notes — measurements that shaped this, and alternatives rejected.
Why not MS-SSIM. multiscale_structural_similarity_index_measure exposes no spatial map, and it refuses inputs below
kernel_size * (len(betas) - 1) ** 2 (176 px for its defaults). Single-scale SSIM does expose a full-resolution differentiable map, so the
pyramid — if used at all — has to be built here. That freedom was then spent deliberately: this module does not reproduce Wang et al.
MS-SSIM. Its contrast-sensitivity levels and fitted betas are a perceptual-quality design tuned against human opinion scores, and a
privacy statistic has no reason to inherit either. In particular MS-SSIM discards the luminance term at fine scales, but a region that
reproduces local mean structure is a leak worth penalizing.
Why the window is small and the stride overlaps. A leak is only reliably detected once it spans roughly twice the window, because a
leak the size of one window splits across the grid unless it happens to be aligned. Measured at 512 px on a synthetic scene with a
saturating cloak, top-1 window score against a no-leak baseline of 0.128: a 32 px leak with window_side_length=32 reads 0.608 grid-aligned but
0.184 unaligned — indistinguishable from the baseline. Overlapping the windows removes the dependence on where the leak falls: sweeping
one leak across the grid, the spread between best and worst position is 1.96x at stride == window, 1.22x at window // 2, and
1.01x at window // 4. Overlap costs no additional similarity evaluation — it is one pooling call over a map that was already computed.
Why the tail is a count and not a fraction. tail_mean is parameterized by a
fraction, which is right for a general reduction: it is dimensionless and keeps its meaning as the position count changes. That reasoning
does not survive here, because the thing being caught is a fixed physical region while the window count grows with image area — so the
invariant is a count and the fraction is the derived quantity. Under the recommended geometry a leak of side L covers exactly
((L - L/2) / (L/4) + 1) ** 2 == 9 windows for every L. Sweeping image sides 256/512/1024 against leaks of 32 and 64 px, a tail of
four windows was the best setting in all six cells and held the leak response flat at 0.81-0.91, while the fraction at that same optimum
ranged over 0.0002-0.0178. Holding the fraction fixed instead degrades sharply with resolution: at 0.005 the response falls from 0.803
to 0.477 to 0.127 at those three image sizes. This module therefore exposes tail_windows and converts internally.
Why num_levels defaults to 1. Across sharp and low-pass leaks at 16/32/64/128 px, no coarser pyramid level detected anything the finest
level missed, including the low-pass case the pyramid was expected to own — SSIM's Gaussian window measures local structural correlation,
which survives blurring, so a coarse-structure leak still registers at full resolution. Coarser levels meanwhile carry a substantially
higher floor, because average-pooling the noisy image partially undoes the cloak's clamp saturation. The parameter is kept so the finding
can be re-tested on real data, not because the default should change.
Why no flat-region gate. Suppressing windows whose clean side is flat looks appealing, because SSIM saturates toward 1 on flat pairs. It
was measured and rejected: clean-side local variance is bimodal (smooth content versus textured), so any threshold that suppresses
smooth-region false positives also deletes smooth-region leaks — and smooth content (faces, skin, sky, document background) is exactly
what is privacy-relevant. With a leak planted inside a smooth region, a threshold of 0.01 on the clean-side standard deviation removed
the leak entirely and dropped the top window score from 0.61 to 0.16. Un-gated, the same leak ranked first of all windows, so the false
positives never outranked it. Clean-side window variance is available as a diagnostic via return_window_diagnostics for callers who want
to check whether a tail is flat-dominated, but it never suppresses a score.
Sensitivity floor on smooth content. There is nonetheless a real limit here, and it is a property of SSIM rather than something this
module can reduce away. SSIM's contrast term is (2 sigma_xy + c2) / (sigma_x^2 + sigma_y^2 + c2) with c2 = (0.03 * data_range) ** 2, so
wherever the clean side's local variance is comparable to or below c2 the stabilizer dominates and the term saturates toward 1 regardless
of what the transform did. Measured on a pure linear ramp (11 px local variance 3.6e-04, below c2 = 9.0e-04), a fully destroyed image
still scored 0.978 and a verbatim leak only lifted it to 0.993 — a usable range of 0.015. On the same ramp plus moderate texture
(local variance 2.7e-02, thirty times c2) the destroyed baseline fell to 0.098 and the leak reached 0.814. So this statistic
discriminates well on content whose local standard deviation clearly exceeds 0.03 and barely at all below it. That is the honest reason
to return clean_window_variances: a caller can see which part of the image the statistic can actually speak about.
Why fp32 is forced. In bfloat16 the SSIM map is not merely imprecise, it leaves its own range: a flat pair at 0.50 versus 0.51 reads
0.999704 in float32 and 2.5625 in bfloat16. The cause is that sigma_pred_target = E[xy] - E[x] E[y] is a catastrophic cancellation
that torchmetrics does not clamp (unlike the two variances). Inputs are therefore promoted rather than trusted.
Classes:
| Name | Description |
|---|---|
WindowedSsimTailResult |
Result of |
Functions:
| Name | Description |
|---|---|
local_normalized_cross_correlation |
Measure how much structure survives a transform, ignoring any per-window brightness and contrast change. |
windowed_ssim_tail_mean |
Score structural similarity per image window_side_length and reduce with a tail, so a localized leak cannot be averaged away. |
Attributes:
| Name | Type | Description |
|---|---|---|
BoundaryPolicy |
What to do when the window grid does not tile a pyramid level exactly. |
|
ChannelReduction |
How a per-channel window score is collapsed to one value. |
|
DEFAULT_BOUNDARY |
Final[BoundaryPolicy]
|
Default boundary policy. See |
DEFAULT_CHANNEL_REDUCTION |
Final[ChannelReduction]
|
Default channel reduction. |
DEFAULT_MIN_VARIANCE |
Final[float]
|
Default variance floor below which a window is excluded from |
DEFAULT_NUM_LEVELS |
Final[int]
|
Default pyramid depth. |
DEFAULT_TAIL_WINDOWS |
Final[int]
|
Default number of worst-scoring windows retained by the tail reduction, per image and per level. |
DEFAULT_WINDOW_SIDE_LENGTH |
Final[int]
|
Default side length of a scoring window, in pixels of the level being scored. |
module-attribute
¶
BoundaryPolicy = Literal['error', 'pad']
What to do when the window grid does not tile a pyramid level exactly.
"error": raiseValueError, leaving the caller to choose a compatible geometry or input size."pad": reflect-pad both images — centered, and identically — up to the smallest size the grid tiles exactly at every level, then score the padded pair.
"pad" exists because the tiling requirement is a joint consequence of window_side_length, stride and num_levels that nothing in the
signature states: under the defaults it reduces to "every side must be a multiple of eight", and it is checked at every pyramid level, so a
side can pass at full resolution and fail once halved. Native- and variable-resolution training does not get to choose its input sizes.
Reflection rather than a constant fill, because a constant border is flat on both sides of the comparison and would score as a perfectly reproduced region — a fabricated leak in exactly the windows the tail selects.
module-attribute
¶
ChannelReduction = Literal['max', 'mean']
How a per-channel window score is collapsed to one value.
"max": the worst channel. A leak confined to a single colour channel reads 0.753 here against 0.269 under"mean", so this is what makes a single-channel leak visible to a vision encoder, which has no chroma subsampling."mean": the channel average. Slightly sharper on leaks that span all channels (measured contrast 0.751 against 0.706) and it does not inflate the floor, which"max"raises by about 0.056 on fully obfuscated content by picking up per-channel noise.
Neither dominates; the choice is which leak shape matters more.
module-attribute
¶
DEFAULT_BOUNDARY: Final[BoundaryPolicy] = 'error'
Default boundary policy. See BoundaryPolicy.
"error" because an untileable grid is usually a misconfigured stride or num_levels rather than an awkward input size, and padding it
away by default would hide that from a caller who could simply fix it. Opting in to "pad" is a statement that the input size is not
under the caller's control.
module-attribute
¶
DEFAULT_CHANNEL_REDUCTION: Final[ChannelReduction] = 'max'
Default channel reduction. "max" because a single-channel leak is otherwise nearly invisible. See ChannelReduction.
The case for keeping "mean" is narrow and it should not be reached for casually. It is sharper only when the leak spans every channel,
where it measured a contrast of 0.751 against 0.706, and it avoids the 0.056 floor inflation "max" picks up from per-channel noise
on fully obfuscated content. Against that, it reports a single-channel leak at 0.269 where "max" reports 0.753. A 6% edge on one leak
shape does not pay for a 2.8x blind spot on another, so "mean" exists for callers who have established that single-channel leaks are
outside their threat model — not as a co-equal alternative.
module-attribute
¶
Default variance floor below which a window is excluded from local_normalized_cross_correlation.
A correlation is undefined without variance, so such windows are dropped instead of scored.
The floor is not a numerical guard. _windowed_centered_moments centers inside each window rather than using the E[x^2] - E[x]^2
identity, so the near-constant regime is computed accurately; what remains is that a ratio of two vanishing quantities carries no
information about how much structure survived. One consequence is worth knowing: because this is a hard threshold, a window sitting within a
few float32 rounding errors of it can flip between scored and excluded under any harmless change to the summation order — a different
stride, device, or chunk size. That is inherent to thresholding, and only reachable by inputs deliberately parked at the floor.
module-attribute
¶
Default pyramid depth. 1 is single-scale, which is what the measurements in the module docstring support.
module-attribute
¶
Default number of worst-scoring windows retained by the tail reduction, per image and per level.
A count, not a fraction, because the quantity being caught is a fixed physical region while the window count grows with image area.
Under the recommended geometry (window_side_length about half the smallest region of interest, stride half the window) a leak of side L covers
exactly ((L - L/2) / (L/4) + 1) ** 2 == 9 windows for every L, so the right count is a constant. Measured across image sides
256/512/1024 and leaks of 32/64 px, 4 was the best setting in every one of the six cells, holding the leak response flat at 0.81-0.91;
the fraction corresponding to that same optimum ranged over 0.0002-0.0178, a hundredfold spread. A fixed fraction instead degrades
sharply with resolution — at 0.005 the response falls 0.803 to 0.477 to 0.127 at those three image sizes.
Values at or above the window count select every window, which is the plain mean.
module-attribute
¶
Default side length of a scoring window, in pixels of the level being scored.
A leak is only reliably detected once it spans roughly twice the window, so this should be at most half the smallest region worth
catching. 16 targets regions of about 32 px and upward — an eye-sized feature at a 512 px input.
Bases: TypedDict
Result of windowed_ssim_tail_mean.
Attributes:
| Name | Type | Description |
|---|---|---|
total |
Tensor
|
Scalar statistic, the mean over pyramid levels and images of each level's windowed tail. This is the loss-ready value. |
level_scores |
NotRequired[Tensor]
|
Per-image, per-level tails of shape |
window_scores |
NotRequired[tuple[Tensor, ...]]
|
One entry per level, each of shape |
clean_window_variances |
NotRequired[tuple[Tensor, ...]]
|
One entry per level, each of shape |
local_normalized_cross_correlation(
noisy_01: Tensor,
clean_01: Tensor,
*,
window_side_length: int = 16,
stride: int | None = None,
channel_reduction: ChannelReduction = "max",
min_variance: float = 1e-06
) -> torch.Tensor
Measure how much structure survives a transform, ignoring any per-window brightness and contrast change.
This is SSIM's structure term on its own — |cov(x, y)| / (sigma_x sigma_y) per window, per channel — with the luminance and
contrast factors dropped. Dropping them is the entire point: both are functions of how much the values were shifted and scaled, and
neither says anything about how much information survived.
Why SSIM alone is not enough. SSIM's contrast factor is (2 sigma_x sigma_y + c2) / (sigma_x^2 + sigma_y^2 + c2), which collapses
toward zero whenever sigma_noisy << sigma_clean. Compressing an image's dynamic range therefore minimizes SSIM while destroying
nothing — the original values are recoverable by rescaling. That is not a hypothetical: a trained checkpoint was shown to be
reversible by a per-channel levels adjustment of six numbers, recovering recognisable faces from images that looked uniformly flat.
Measured on transforms that are exactly invertible, so a correct statistic must report 1.0:
| transform (information-preserving) | ms_ssim_total |
this statistic |
|---|---|---|
affine squash into [0.96, 1.0] |
0.7804 | 1.0000 |
| red/green compressed, blue scaled to zero | 0.7204 | 1.0000 |
| independent noise (does destroy) | 0.2269 | 0.0489 |
Because a correlation is invariant to any affine map of either argument, no amount of shifting, scaling or channel-wise rescaling can
lower this score. The transform has to actually decorrelate the content. The two invertible rows are exact, not rounded: an affine remap
leaves every window's correlation at one identically, so 1.0000 is the arithmetic reproducing a mathematical certainty rather than a
close call. Earlier revisions of this table read 1.0038 and 1.0016, which were measuring the second-moment error since removed —
see _windowed_centered_moments.
Locality is what makes it strong. A global correlation is defeated by making the compression vary across the image — measured, a 2% spatial modulation is enough to collapse a global attacker view to its noise floor. Computing the correlation per window means the invariance holds per window, so a spatially varying remap is still seen. The window should be smaller than the spatial scale over which the transform varies; too large and the per-window affine assumption breaks down.
Windows whose clean or noisy side is nearly constant are excluded, not scored: a correlation is undefined without variance, and a ratio
of two vanishing quantities says nothing about how much structure survived. See DEFAULT_MIN_VARIANCE.
Privacy: computed from user-provided image data. The returned scalar is an aggregate and safe to log.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
Transformed image tensor of shape |
required |
|
Tensor
|
Clean image tensor of the same shape. Detached internally, so gradients reach only |
required |
|
int
|
Side length of the correlation window, in pixels. Should be smaller than the scale over which the transform varies. |
16
|
|
int | None
|
Window stride. |
None
|
|
ChannelReduction
|
How to collapse per-channel correlations within a window. See |
'max'
|
|
float
|
Windows whose clean or noisy variance falls below this are excluded from the mean. Guards the undefined and numerically unstable near-constant regime. |
1e-06
|
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
Scalar in |
torch.Tensor
|
more residual structure, so this is minimized as a privacy penalty. Always |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the inputs are not matching 4-D tensors, if |
Example
torch.manual_seed(0) # doctest: +ELLIPSIS
clean = torch.rand(1, 3, 64, 64) Squashing the range destroys nothing, and this statistic says so.¶
squashed = clean * 0.04 + 0.96 bool(local_normalized_cross_correlation(squashed, clean) > 0.95) True
Independent noise really does destroy it.¶
bool(local_normalized_cross_correlation(torch.rand(1, 3, 64, 64), clean) < 0.3) True
Added in version v3.71.0. Windowed correlation privacy statistic that is invariant to a per-window affine remap, closing the gap that lets a range-compressing transform read as private under SSIM.
windowed_ssim_tail_mean(
noisy_01: Tensor,
clean_01: Tensor,
*,
window_side_length: int = 16,
stride: int | None = None,
num_levels: int = 1,
tail_windows: int = 4,
channel_reduction: ChannelReduction = "max",
boundary: BoundaryPolicy = "error",
return_level_scores: bool = False,
return_window_diagnostics: bool = False
) -> WindowedSsimTailResult
Score structural similarity per image window_side_length and reduce with a tail, so a localized leak cannot be averaged away.
Both inputs must already be in the [0, 1] range — denormalize from any model normalization before calling. Higher values mean
more residual similarity, so this is minimized as a privacy penalty and composes directly with
squared_hinge_loss (which is elementwise, and so also applies to
level_scores or a window_scores entry).
For each pyramid level the SSIM map is taken at full resolution, average-pooled into windows, collapsed across channels, and the
tail_windows worst windows are averaged by tail_mean. Levels are then combined with an
unweighted mean; return_level_scores exposes the per-level values so a caller can weight them differently. The image is
average-pooled by two between levels, while window_side_length stays fixed in pixels of the level being scored — so a window covers
window * 2 ** level pixels of the original image, and the similarity kernel keeps a constant ratio to the window at every level.
Inputs are promoted to float32 and the result is float32 regardless of input dtype. This is not a convenience: in
bfloat16 the similarity map leaves its own range (a flat pair at 0.50 versus 0.51 reads 2.5625 rather than 0.999704), because
the covariance term is a catastrophic cancellation that torchmetrics does not clamp.
No NaN or infinity guard is applied. A non-finite value in noisy_01 propagates to total, deliberately: this is a primitive and
the surrounding objective owns that policy, matching
tail_mean. Callers wanting the sanitizing behavior of
safe_ms_ssim should apply it themselves.
Privacy: this statistic is computed from user-provided image data. total and level_scores are aggregates and safe to log, but
the tensors returned under return_window_diagnostics are spatially resolved and per-region — treat them as sensitive and do not
log or persist them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
Transformed image tensor of shape |
required |
|
Tensor
|
Clean image tensor of the same shape, with values in |
required |
|
int
|
Window side length, in pixels of the level being scored. Should be at most half the smallest region worth catching, since
a leak spanning only one window can split across the grid. See |
16
|
|
int | None
|
Window stride. |
None
|
|
int
|
Pyramid depth. |
1
|
|
int
|
Number of worst-scoring windows averaged per image and per level. A count rather than a fraction, because
the region being caught is a fixed physical size while the window count grows with image area — so the count transfers
across resolutions and a fraction does not. A value at or above the window count selects them all, which is the plain mean
and reproduces the dilution this function exists to avoid. See |
4
|
|
ChannelReduction
|
How to collapse per-channel window scores. See |
'max'
|
|
BoundaryPolicy
|
What to do when the window grid does not tile a pyramid level exactly. |
'error'
|
|
bool
|
Also return the per-image, per-level tails. |
False
|
|
bool
|
Also return every window's score and the clean side's per-window variance, per level. |
False
|
Returns:
| Type | Description |
|---|---|
WindowedSsimTailResult
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the inputs are not matching 4-D tensors, if |
Example
clean = ( ... torch.linspace(0.0, 1.0, 64 * 64) ... .reshape(1, 1, 64, 64) ... .expand(1, 3, 64, 64) ... .contiguous() ... )
A transform that destroys the image everywhere except one 32x32 corner.¶
noisy = (clean + 3.0).clamp(0.0, 1.0) noisy[..., :32, :32] = clean[..., :32, :32] leaked = windowed_ssim_tail_mean(noisy, clean, window_side_length=16)["total"] destroyed = windowed_ssim_tail_mean( ... (clean + 3.0).clamp(0.0, 1.0), clean, window_side_length=16 ... )["total"] bool(leaked > destroyed) True
A plain mean over windows dilutes the same leak away.¶
pooled_leaked = windowed_ssim_tail_mean( ... noisy, clean, window_side_length=16, tail_windows=10**6 ... )["total"] bool(leaked - destroyed > pooled_leaked - destroyed) True
level_scoresreproducestotal.¶result = windowed_ssim_tail_mean( ... noisy, clean, window_side_length=16, num_levels=2, return_level_scores=True ... ) result["level_scores"].shape torch.Size([1, 2]) bool(torch.allclose(result["level_scores"].mean(), result["total"])) True
63 px is not tiled by the default grid, so scoring it at all takes an explicit boundary policy.¶
cropped = clean[..., :63, :63].contiguous() padded = windowed_ssim_tail_mean( ... cropped, cropped, window_side_length=16, boundary="pad" ... ) bool(padded["total"] > 0.99) True
Added in version v3.71.0. The `boundary` argument, which lets a caller reflect-pad an input the window grid does not tile exactly rather than take a `ValueError`. The default is unchanged.
Added in version v3.71.0. Spatially-localized structural-similarity privacy statistic. Scores every (optionally overlapping) window_side_length of the image and reduces with a tail, so a single surviving region sets the value instead of being averaged away as it is by `multi_attacker_privacy_loss`.