interpolation
Module containing the selectable interpolation controls shared by the vision Stained Glass Transform noise layers.
The vision cloaks resample tensors at the estimator boundary, where the input is downscaled to a fixed internal resolution
before the backbone runs and the raw estimate is upscaled back to the native resolution afterwards. Both legs go through
resize_boundary, and
round_trip composes the two to band-limit a tensor to the internal
resolution without changing its shape.
The backend is fixed at torchvision's resize (which antialiases when downsampling with a linear mode), because that is what
the boundary historically used and the noise layers must keep reproducing prior releases bit-for-bit. Only the interpolation
mode is selectable.
Classes:
| Name | Description |
|---|---|
ComposedResizer |
Apply any number of |
InputResizer |
Strategy for resizing a tensor at an estimator boundary, and reversing that resize afterward. |
InterpolationMode |
Interpolation mode selectable for the vision noise layers' resampling steps. |
NoResize |
No resizing at all: run the estimator module at the native input resolution. |
PadToMultipleOf |
Pad each spatial dimension up to a multiple of |
ResampleToFixedSize |
Resize to a fixed |
ResizeToMultipleOf |
Resize each spatial dimension to the nearest multiple of |
Functions:
| Name | Description |
|---|---|
resize_boundary |
Resize a tensor across an estimator boundary. |
resizer_from_config |
Build an |
resolve |
Normalize an interpolation mode argument to an |
round_trip |
Resample a tensor down to an internal resolution and back up to its original spatial size. |
validate_cloak_modes |
Validate a pair of estimator boundary interpolation modes for a noise layer. |
validate_estimator_modes |
Validate that a pair of estimator boundary interpolation modes is coherent. |
validate_matching_estimator_controls |
Validate that mean and standard deviation estimators have matching cloak controls. |
ComposedResizer
¶
Bases: InputResizer
Apply any number of InputResizer stages in sequence.
Methods:
| Name | Description |
|---|---|
__init__ |
Initialize the resizer. |
resamples_input |
Whether any stage's |
resize |
Resize |
resize_back |
Resize |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict. |
__init__
¶
__init__(*stages: InputResizer) -> None
Initialize the resizer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InputResizer
|
The resizers to apply, in order, on the way in; in reverse order, on the way back. Must be non-empty. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
resamples_input
¶
Whether any stage's skip_connection_input band-limits its input.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
¶
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
to_config
¶
Serialize this resizer to a JSON-safe config dict.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
InputResizer
¶
Bases: ABC
Strategy for resizing a tensor at an estimator boundary, and reversing that resize afterward.
Methods:
| Name | Description |
|---|---|
resamples_input |
Whether |
resize |
Resize |
resize_back |
Resize |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict, invertible by |
resamples_input
abstractmethod
¶
Whether skip_connection_input band-limits its input.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
abstractmethod
¶
resize_back
abstractmethod
¶
resize_back(
output: Tensor, original_size: Sequence[int]
) -> torch.Tensor
Resize output back to original_size, undoing resize, or return it unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to resize back. |
required |
|
Sequence[int]
|
The |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The resized tensor. |
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
to_config
abstractmethod
¶
Serialize this resizer to a JSON-safe config dict, invertible by resizer_from_config.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A dict with a |
InterpolationMode
¶
Bases: StrEnum
Interpolation mode selectable for the vision noise layers' resampling steps.
Each member's value is accepted verbatim by torchvision.transforms.v2.InterpolationMode, which is the backend every
resampling site dispatches to.
NEAREST is deliberately absent. It is the legacy nearest-neighbor implementation, whose off-by-one sampling grid makes it
inconsistent with every other mode; NEAREST_EXACT is its corrected counterpart and should be used instead.
Added in version v3.64.0.
NoResize
¶
Bases: InputResizer
No resizing at all: run the estimator module at the native input resolution.
Methods:
| Name | Description |
|---|---|
resamples_input |
Whether |
resize |
Return |
resize_back |
Return |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict. |
resamples_input
¶
Whether skip_connection_input band-limits its input.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
¶
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
to_config
¶
Serialize this resizer to a JSON-safe config dict.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
PadToMultipleOf
¶
Bases: InputResizer
Pad each spatial dimension up to a multiple of multiple, and crop back afterward.
Unlike ResizeToMultipleOf, no interpolation is involved: the input keeps its native pixel grid and is padded on the
bottom and right, so the estimate is cropped back to exactly the pixels the input occupied.
Methods:
| Name | Description |
|---|---|
__init__ |
Initialize the resizer. |
resamples_input |
Whether |
resize |
Pad |
resize_back |
Crop |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict. |
__init__
¶
resamples_input
¶
Whether skip_connection_input band-limits its input.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
¶
Pad input on the bottom and right up to a multiple of self.multiple.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to pad. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The padded tensor, or |
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
to_config
¶
Serialize this resizer to a JSON-safe config dict.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
ResampleToFixedSize
¶
Bases: InputResizer
Resize to a fixed (height, width), e.g. MultiScaleVisionTransformerCloak's INTERNAL_TRANSFORMER_IMG_SIZE grid.
Methods:
| Name | Description |
|---|---|
__init__ |
Initialize the resizer. |
resamples_input |
Whether |
resize |
Resize |
resize_back |
Resize |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict. |
__init__
¶
__init__(
size: tuple[int, int],
pre_mode: InterpolationMode,
post_mode: InterpolationMode | None,
resample_input: bool,
) -> None
Initialize the resizer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
tuple[int, int]
|
The fixed |
required |
|
InterpolationMode
|
The interpolation mode used to resize down to |
required |
|
InterpolationMode | None
|
The interpolation mode used to resize back up to the native resolution, or |
required |
|
bool
|
Whether |
required |
resamples_input
¶
Whether skip_connection_input band-limits its input via round_trip.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
¶
resize_back
¶
resize_back(
output: Tensor, original_size: Sequence[int]
) -> torch.Tensor
Resize output back up to original_size, or leave it at self.size if self.post_mode is None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to resize back. |
required |
|
Sequence[int]
|
The |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The resized tensor, or |
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
to_config
¶
Serialize this resizer to a JSON-safe config dict.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
|
ResizeToMultipleOf
¶
Bases: InputResizer
Resize each spatial dimension to the nearest multiple of multiple, at minimum multiple itself.
Methods:
| Name | Description |
|---|---|
__init__ |
Initialize the resizer. |
resamples_input |
Whether |
resize |
Resize |
resize_back |
Resize |
round_trip |
Resample |
skip_connection_input |
Return the tensor the sampled noise should be added to. |
to_config |
Serialize this resizer to a JSON-safe config dict. |
__init__
¶
__init__(
pre_mode: InterpolationMode,
post_mode: InterpolationMode | None,
resample_input: bool,
multiple: int = 32,
) -> None
Initialize the resizer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InterpolationMode
|
The interpolation mode used to resize down to the nearest multiple of |
required |
|
InterpolationMode | None
|
The interpolation mode used to resize back up to the native resolution, or |
required |
|
bool
|
Whether |
required |
|
int
|
The multiple each spatial dimension is rounded to. |
32
|
resamples_input
¶
Whether skip_connection_input band-limits its input via round_trip.
Returns:
| Type | Description |
|---|---|
bool
|
|
resize
¶
resize_back
¶
resize_back(
output: Tensor, original_size: Sequence[int]
) -> torch.Tensor
Resize output back up to original_size, or leave it at the nearest multiple of self.multiple if
self.post_mode is None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to resize back. |
required |
|
Sequence[int]
|
The |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The resized tensor, or |
round_trip
¶
Resample input down, then back up unless resize_back is configured to skip that leg.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
torch.Tensor
|
|
torch.Tensor
|
|
skip_connection_input
¶
Return the tensor the sampled noise should be added to.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The original, untransformed input. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
|
torch.Tensor
|
|
to_config
¶
Serialize this resizer to a JSON-safe config dict.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
`{"type": "resize_to_multiple_of", "pre_mode": self.pre_mode, "post_mode": self.post_mode, |
dict[str, Any]
|
"resample_input": self.resample_input, "multiple": self.multiple}`. |
resize_boundary
¶
Resize a tensor across an estimator boundary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to resize. The trailing two dimensions are treated as the spatial dimensions. |
required |
|
Sequence[int]
|
The target |
required |
|
InterpolationMode
|
The interpolation mode to use. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The resized tensor. |
Added in version v3.64.0.
resizer_from_config
¶
Build an InputResizer from either the old pre/post_estimator_interpolation fields or a new resizer field.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Any
|
Either a |
required |
Returns:
| Type | Description |
|---|---|
InputResizer
|
The constructed |
resolve
¶
resolve(
value: InterpolationMode | str | None,
) -> InterpolationMode | None
Normalize an interpolation mode argument to an InterpolationMode member, or None for no interpolation.
Constructors call this so that the attribute they store, and therefore the value they serialize, is always a canonical member rather than an equivalent string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InterpolationMode | str | None
|
An |
required |
Returns:
| Type | Description |
|---|---|
InterpolationMode | None
|
The corresponding |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Added in version v3.64.0.
round_trip
¶
round_trip(
input: Tensor,
internal_size: Sequence[int],
pre: InterpolationMode,
post: InterpolationMode,
) -> torch.Tensor
Resample a tensor down to an internal resolution and back up to its original spatial size.
The result has the same shape as input but is band-limited to internal_size, which is how the vision noise layers
band-limit the skip connection to match the resolution their estimates were computed at.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Tensor
|
The tensor to round-trip. The trailing two dimensions are treated as the spatial dimensions. |
required |
|
Sequence[int]
|
The intermediate |
required |
|
InterpolationMode
|
The interpolation mode used on the way down to |
required |
|
InterpolationMode
|
The interpolation mode used on the way back up to the original size. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The round-tripped tensor, with the same shape as |
Added in version v3.64.0.
validate_cloak_modes
¶
validate_cloak_modes(
pre: InterpolationMode | None,
post: InterpolationMode | None,
) -> None
Validate a pair of estimator boundary interpolation modes for a noise layer.
Stricter than validate_estimator_modes: an estimator
on its own may return an estimate at the internal resolution, but a noise layer composes input + std * noise + mean at the
native input resolution, so the estimates have to come back. The two modes must therefore both be set or both be None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InterpolationMode | None
|
The mode used to interpolate the estimator input down to the internal resolution, or |
required |
|
InterpolationMode | None
|
The mode used to interpolate the raw estimate back to the native resolution, or |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If exactly one of |
Added in version v3.64.0.
validate_estimator_modes
¶
validate_estimator_modes(
pre: InterpolationMode | None,
post: InterpolationMode | None,
) -> None
Validate that a pair of estimator boundary interpolation modes is coherent.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InterpolationMode | None
|
The mode used to interpolate the estimator input down to the internal resolution, or |
required |
|
InterpolationMode | None
|
The mode used to interpolate the raw estimate back to the native resolution, or |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Added in version v3.64.0.
validate_matching_estimator_controls
¶
validate_matching_estimator_controls(
mean_pre: InterpolationMode | None,
mean_post: InterpolationMode | None,
std_pre: InterpolationMode | None,
std_post: InterpolationMode | None,
) -> None
Validate that mean and standard deviation estimators have matching cloak controls.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
InterpolationMode | None
|
The mean estimator's pre-estimator interpolation mode. |
required |
|
InterpolationMode | None
|
The mean estimator's post-estimator interpolation mode. |
required |
|
InterpolationMode | None
|
The standard deviation estimator's pre-estimator interpolation mode. |
required |
|
InterpolationMode | None
|
The standard deviation estimator's post-estimator interpolation mode. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If any interpolation control differs between the estimators. |