Skip to content

interpolation

Module containing the selectable interpolation controls shared by the vision Stained Glass Transform noise layers.

The vision cloaks resample tensors at the estimator boundary, where the input is downscaled to a fixed internal resolution before the backbone runs and the raw estimate is upscaled back to the native resolution afterwards. Both legs go through resize_boundary, and round_trip composes the two to band-limit a tensor to the internal resolution without changing its shape.

The backend is fixed at torchvision's resize (which antialiases when downsampling with a linear mode), because that is what the boundary historically used and the noise layers must keep reproducing prior releases bit-for-bit. Only the interpolation mode is selectable.

Classes:

Name Description
ComposedResizer

Apply any number of InputResizer stages in sequence.

InputResizer

Strategy for resizing a tensor at an estimator boundary, and reversing that resize afterward.

InterpolationMode

Interpolation mode selectable for the vision noise layers' resampling steps.

NoResize

No resizing at all: run the estimator module at the native input resolution.

PadToMultipleOf

Pad each spatial dimension up to a multiple of multiple, and crop back afterward.

ResampleToFixedSize

Resize to a fixed (height, width), e.g. MultiScaleVisionTransformerCloak's INTERNAL_TRANSFORMER_IMG_SIZE grid.

ResizeToMultipleOf

Resize each spatial dimension to the nearest multiple of multiple, at minimum multiple itself.

Functions:

Name Description
resize_boundary

Resize a tensor across an estimator boundary.

resizer_from_config

Build an InputResizer from either the old pre/post_estimator_interpolation fields or a new resizer field.

resolve

Normalize an interpolation mode argument to an InterpolationMode member, or None for no interpolation.

round_trip

Resample a tensor down to an internal resolution and back up to its original spatial size.

validate_cloak_modes

Validate a pair of estimator boundary interpolation modes for a noise layer.

validate_estimator_modes

Validate that a pair of estimator boundary interpolation modes is coherent.

validate_matching_estimator_controls

Validate that mean and standard deviation estimators have matching cloak controls.

ComposedResizer

Bases: InputResizer

Apply any number of InputResizer stages in sequence.

Methods:

Name Description
__init__

Initialize the resizer.

resamples_input

Whether any stage's skip_connection_input band-limits its input.

resize

Resize input via each stage in self.stages, in order.

resize_back

Resize output back via each stage in self.stages, in reverse order.

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict.

__init__

__init__(*stages: InputResizer) -> None

Initialize the resizer.

Parameters:

Name Type Description Default

*stages

InputResizer

The resizers to apply, in order, on the way in; in reverse order, on the way back. Must be non-empty.

required

Raises:

Type Description
ValueError

If stages is empty.

resamples_input

resamples_input() -> bool

Whether any stage's skip_connection_input band-limits its input.

Returns:

Type Description
bool

True if any stage's resamples_input() is True, False otherwise.

resize

resize(input: Tensor) -> torch.Tensor

Resize input via each stage in self.stages, in order.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

resize_back

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Resize output back via each stage in self.stages, in reverse order.

Parameters:

Name Type Description Default

output

Tensor

The tensor to resize back.

required

original_size

Sequence[int]

The (height, width) to resize back to.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input passed through each stage's skip_connection_input, in order.

to_config

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict.

Returns:

Type Description
dict[str, Any]

{"type": "composed", "stages": [stage.to_config() for stage in self.stages]}.

InputResizer

Bases: ABC

Strategy for resizing a tensor at an estimator boundary, and reversing that resize afterward.

Methods:

Name Description
resamples_input

Whether skip_connection_input band-limits its input.

resize

Resize input according to this strategy, or return it unchanged.

resize_back

Resize output back to original_size, undoing resize, or return it unchanged.

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict, invertible by resizer_from_config.

resamples_input abstractmethod

resamples_input() -> bool

Whether skip_connection_input band-limits its input.

Returns:

Type Description
bool

True if skip_connection_input band-limits its input, False otherwise.

resize abstractmethod

resize(input: Tensor) -> torch.Tensor

Resize input according to this strategy, or return it unchanged.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

resize_back abstractmethod

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Resize output back to original_size, undoing resize, or return it unchanged.

Parameters:

Name Type Description Default

output

Tensor

The tensor to resize back.

required

original_size

Sequence[int]

The (height, width) to resize back to.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input, by default. Subclasses may override this to band-limit the skip connection.

to_config abstractmethod

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict, invertible by resizer_from_config.

Returns:

Type Description
dict[str, Any]

A dict with a "type" key identifying the concrete class, plus that class's constructor arguments.

InterpolationMode

Bases: StrEnum

Interpolation mode selectable for the vision noise layers' resampling steps.

Each member's value is accepted verbatim by torchvision.transforms.v2.InterpolationMode, which is the backend every resampling site dispatches to.

NEAREST is deliberately absent. It is the legacy nearest-neighbor implementation, whose off-by-one sampling grid makes it inconsistent with every other mode; NEAREST_EXACT is its corrected counterpart and should be used instead.

Added in version v3.64.0.

NoResize

Bases: InputResizer

No resizing at all: run the estimator module at the native input resolution.

Methods:

Name Description
resamples_input

Whether skip_connection_input band-limits its input.

resize

Return input unchanged.

resize_back

Return output unchanged.

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict.

resamples_input

resamples_input() -> bool

Whether skip_connection_input band-limits its input.

Returns:

Type Description
bool

False: there is no resizing at all, so skip_connection_input never band-limits anything.

resize

resize(input: Tensor) -> torch.Tensor

Return input unchanged.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize.

required

Returns:

Type Description
torch.Tensor

input, unchanged.

resize_back

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Return output unchanged.

Parameters:

Name Type Description Default

output

Tensor

The tensor to resize back.

required

original_size

Sequence[int]

Ignored - output is already at the native resolution.

required

Returns:

Type Description
torch.Tensor

output, unchanged.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input, by default. Subclasses may override this to band-limit the skip connection.

to_config

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict.

Returns:

Type Description
dict[str, Any]

{"type": "no_resize"}.

PadToMultipleOf

Bases: InputResizer

Pad each spatial dimension up to a multiple of multiple, and crop back afterward.

Unlike ResizeToMultipleOf, no interpolation is involved: the input keeps its native pixel grid and is padded on the bottom and right, so the estimate is cropped back to exactly the pixels the input occupied.

Methods:

Name Description
__init__

Initialize the resizer.

resamples_input

Whether skip_connection_input band-limits its input.

resize

Pad input on the bottom and right up to a multiple of self.multiple.

resize_back

Crop output back to original_size, discarding the padded region.

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict.

__init__

__init__(
    multiple: int = 32, pad_value: float = 0.0
) -> None

Initialize the resizer.

Parameters:

Name Type Description Default

multiple

int

The multiple each spatial dimension is padded up to.

32

pad_value

float

The constant value used for the padded region.

0.0

resamples_input

resamples_input() -> bool

Whether skip_connection_input band-limits its input.

Returns:

Type Description
bool

False: padding and cropping preserve the native pixel grid exactly.

resize

resize(input: Tensor) -> torch.Tensor

Pad input on the bottom and right up to a multiple of self.multiple.

Parameters:

Name Type Description Default

input

Tensor

The tensor to pad. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The padded tensor, or input unchanged if both spatial dimensions are already multiples of self.multiple.

resize_back

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Crop output back to original_size, discarding the padded region.

Parameters:

Name Type Description Default

output

Tensor

The tensor to crop.

required

original_size

Sequence[int]

The (height, width) to crop back to.

required

Returns:

Type Description
torch.Tensor

The cropped tensor.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input, by default. Subclasses may override this to band-limit the skip connection.

to_config

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict.

Returns:

Type Description
dict[str, Any]

{"type": "pad_to_multiple_of", "multiple": self.multiple, "pad_value": self.pad_value}.

ResampleToFixedSize

Bases: InputResizer

Resize to a fixed (height, width), e.g. MultiScaleVisionTransformerCloak's INTERNAL_TRANSFORMER_IMG_SIZE grid.

Methods:

Name Description
__init__

Initialize the resizer.

resamples_input

Whether skip_connection_input band-limits its input via round_trip.

resize

Resize input down to self.size.

resize_back

Resize output back up to original_size, or leave it at self.size if self.post_mode is None.

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict.

__init__

Initialize the resizer.

Parameters:

Name Type Description Default

size

tuple[int, int]

The fixed (height, width) to resize to.

required

pre_mode

InterpolationMode

The interpolation mode used to resize down to size.

required

post_mode

InterpolationMode | None

The interpolation mode used to resize back up to the native resolution, or None to skip that resize entirely and leave the output at size. With resample_input=True, this also leaves the skip connection at size rather than the native resolution, so the noise layer composes entirely at size (mean, std, and the skip term all at size) instead of upscaling anything back to native.

required

resample_input

bool

Whether skip_connection_input should band-limit its input via round_trip.

required

resamples_input

resamples_input() -> bool

Whether skip_connection_input band-limits its input via round_trip.

Returns:

Type Description
bool

self.resample_input.

resize

resize(input: Tensor) -> torch.Tensor

Resize input down to self.size.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

resize_back

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Resize output back up to original_size, or leave it at self.size if self.post_mode is None.

Parameters:

Name Type Description Default

output

Tensor

The tensor to resize back.

required

original_size

Sequence[int]

The (height, width) to resize back to.

required

Returns:

Type Description
torch.Tensor

The resized tensor, or output unchanged if self.post_mode is None.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input round-tripped through self.size if self.resample_input is set, input unchanged otherwise.

to_config

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict.

Returns:

Type Description
dict[str, Any]

{"type": "resize_to", "size": self.size, "pre_mode": self.pre_mode, "post_mode": self.post_mode, "resample_input": self.resample_input}.

ResizeToMultipleOf

Bases: InputResizer

Resize each spatial dimension to the nearest multiple of multiple, at minimum multiple itself.

Methods:

Name Description
__init__

Initialize the resizer.

resamples_input

Whether skip_connection_input band-limits its input via round_trip.

resize

Resize input down to the nearest multiple of self.multiple.

resize_back

Resize output back up to original_size, or leave it at the nearest multiple of self.multiple if

round_trip

Resample input down, then back up unless resize_back is configured to skip that leg.

skip_connection_input

Return the tensor the sampled noise should be added to.

to_config

Serialize this resizer to a JSON-safe config dict.

__init__

Initialize the resizer.

Parameters:

Name Type Description Default

pre_mode

InterpolationMode

The interpolation mode used to resize down to the nearest multiple of multiple.

required

post_mode

InterpolationMode | None

The interpolation mode used to resize back up to the native resolution, or None to skip that resize entirely and leave the output at the nearest multiple of multiple. With resample_input=True, this also leaves the skip connection at that reduced size rather than the native resolution, so the noise layer composes entirely at the reduced size instead of upscaling anything back to native.

required

resample_input

bool

Whether skip_connection_input should band-limit its input via round_trip.

required

multiple

int

The multiple each spatial dimension is rounded to.

32

resamples_input

resamples_input() -> bool

Whether skip_connection_input band-limits its input via round_trip.

Returns:

Type Description
bool

self.resample_input.

resize

resize(input: Tensor) -> torch.Tensor

Resize input down to the nearest multiple of self.multiple.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

resize_back

resize_back(
    output: Tensor, original_size: Sequence[int]
) -> torch.Tensor

Resize output back up to original_size, or leave it at the nearest multiple of self.multiple if self.post_mode is None.

Parameters:

Name Type Description Default

output

Tensor

The tensor to resize back.

required

original_size

Sequence[int]

The (height, width) to resize back to.

required

Returns:

Type Description
torch.Tensor

The resized tensor, or output unchanged if self.post_mode is None.

round_trip

round_trip(input: Tensor) -> torch.Tensor

Resample input down, then back up unless resize_back is configured to skip that leg.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input if resize_back resizes back to

torch.Tensor

original_size, or resize's reduced shape if resize_back is a no-op (e.g. ResampleToFixedSize/

torch.Tensor

ResizeToMultipleOf with post_mode=None).

skip_connection_input

skip_connection_input(input: Tensor) -> torch.Tensor

Return the tensor the sampled noise should be added to.

Parameters:

Name Type Description Default

input

Tensor

The original, untransformed input.

required

Returns:

Type Description
torch.Tensor

input round-tripped through the nearest multiple of self.multiple if self.resample_input is set,

torch.Tensor

input unchanged otherwise.

to_config

to_config() -> dict[str, Any]

Serialize this resizer to a JSON-safe config dict.

Returns:

Type Description
dict[str, Any]

`{"type": "resize_to_multiple_of", "pre_mode": self.pre_mode, "post_mode": self.post_mode,

dict[str, Any]

"resample_input": self.resample_input, "multiple": self.multiple}`.

resize_boundary

resize_boundary(
    input: Tensor,
    size: Sequence[int],
    mode: InterpolationMode,
) -> torch.Tensor

Resize a tensor across an estimator boundary.

Parameters:

Name Type Description Default

input

Tensor

The tensor to resize. The trailing two dimensions are treated as the spatial dimensions.

required

size

Sequence[int]

The target (height, width).

required

mode

InterpolationMode

The interpolation mode to use.

required

Returns:

Type Description
torch.Tensor

The resized tensor.

Added in version v3.64.0.

resizer_from_config

resizer_from_config(**kwargs: Any) -> InputResizer

Build an InputResizer from either the old pre/post_estimator_interpolation fields or a new resizer field.

Parameters:

Name Type Description Default

**kwargs

Any

Either a resizer mapping (new structure, shaped like InputResizer.to_config()'s output), or pre_estimator_interpolation / post_estimator_interpolation / resample_input (old structure).

required

Returns:

Type Description
InputResizer

The constructed InputResizer.

resolve

resolve(
    value: InterpolationMode | str | None,
) -> InterpolationMode | None

Normalize an interpolation mode argument to an InterpolationMode member, or None for no interpolation.

Constructors call this so that the attribute they store, and therefore the value they serialize, is always a canonical member rather than an equivalent string.

Parameters:

Name Type Description Default

value

InterpolationMode | str | None

An InterpolationMode member, one of its string values, or None for no interpolation.

required

Returns:

Type Description
InterpolationMode | None

The corresponding InterpolationMode member, or None if value is None.

Raises:

Type Description
ValueError

If value is neither None nor a valid InterpolationMode.

Added in version v3.64.0.

round_trip

Resample a tensor down to an internal resolution and back up to its original spatial size.

The result has the same shape as input but is band-limited to internal_size, which is how the vision noise layers band-limit the skip connection to match the resolution their estimates were computed at.

Parameters:

Name Type Description Default

input

Tensor

The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions.

required

internal_size

Sequence[int]

The intermediate (height, width) to resample through.

required

pre

InterpolationMode

The interpolation mode used on the way down to internal_size.

required

post

InterpolationMode

The interpolation mode used on the way back up to the original size.

required

Returns:

Type Description
torch.Tensor

The round-tripped tensor, with the same shape as input.

Added in version v3.64.0.

validate_cloak_modes

validate_cloak_modes(
    pre: InterpolationMode | None,
    post: InterpolationMode | None,
) -> None

Validate a pair of estimator boundary interpolation modes for a noise layer.

Stricter than validate_estimator_modes: an estimator on its own may return an estimate at the internal resolution, but a noise layer composes input + std * noise + mean at the native input resolution, so the estimates have to come back. The two modes must therefore both be set or both be None.

Parameters:

Name Type Description Default

pre

InterpolationMode | None

The mode used to interpolate the estimator input down to the internal resolution, or None for no interpolation.

required

post

InterpolationMode | None

The mode used to interpolate the raw estimate back to the native resolution, or None for no interpolation.

required

Raises:

Type Description
ValueError

If exactly one of pre and post is None.

Added in version v3.64.0.

validate_estimator_modes

validate_estimator_modes(
    pre: InterpolationMode | None,
    post: InterpolationMode | None,
) -> None

Validate that a pair of estimator boundary interpolation modes is coherent.

Parameters:

Name Type Description Default

pre

InterpolationMode | None

The mode used to interpolate the estimator input down to the internal resolution, or None for no interpolation.

required

post

InterpolationMode | None

The mode used to interpolate the raw estimate back to the native resolution, or None for no interpolation.

required

Raises:

Type Description
ValueError

If pre is None while post is not.

Added in version v3.64.0.

validate_matching_estimator_controls

validate_matching_estimator_controls(
    mean_pre: InterpolationMode | None,
    mean_post: InterpolationMode | None,
    std_pre: InterpolationMode | None,
    std_post: InterpolationMode | None,
) -> None

Validate that mean and standard deviation estimators have matching cloak controls.

Parameters:

Name Type Description Default

mean_pre

InterpolationMode | None

The mean estimator's pre-estimator interpolation mode.

required

mean_post

InterpolationMode | None

The mean estimator's post-estimator interpolation mode.

required

std_pre

InterpolationMode | None

The standard deviation estimator's pre-estimator interpolation mode.

required

std_post

InterpolationMode | None

The standard deviation estimator's post-estimator interpolation mode.

required

Raises:

Type Description
ValueError

If any interpolation control differs between the estimators.