Skip to content

diffusion_cloak

Module for Diffusion Cloak noise layers.

Classes:

Name Description
DiffusionCloak

Applies a stochastic transformation to a causal language model input embedding Tensor using TransformerBlockEstimator,

DiffusionCloak

Bases: TransformerCloak

Applies a stochastic transformation to a causal language model input embedding Tensor using TransformerBlockEstimator, with standard deviations parameterized by either CloakStandardDeviationParameterization or DirectStandardDeviationParameterization.

Uses diffusion to explore the input embedding space more deeply for stronger obfuscations. Approximating solutions to stochastic differential equations allows DiffusionCloak to learn a more complicated distribution of inputs that the causal language model treats similarly to the original, untransformed inputs.

Parameters:

Name Type Description Default

*args

Any

Positional arguments used to define the transformer parameter models.

required

num_diffusion_steps

int

The number of steps to use in generating the transformation.

11

stopping_time

float

The final time of the diffusion.

1.0

post_processor

PostProcessor | None

An optional PostProcessor to apply to the diffused output. See BaseNoiseLayer.

None

**kwargs

Any

Keyword arguments used to define the transformer parameter models.

required

Raises:

Type Description
NotImplementedError

If percent_to_mask is not None.

ValueError

If num_diffusion_steps is not positive.

ValueError

If stopping_time is not positive.

Note

For more information on SDE SGT, Ito diffusion, and the numerical approximation of SDE's see: * https://en.wikipedia.org/wiki/Stochastic_differential_equation * https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_method_(SDE)

Changed in version v2.22.0: Removed deepspeed mixture of experts support from transformer cloak.

Methods:

Name Description
__call__

Transform the input data.

__getstate__

Prepare a JSON-serializable copy of the noise layer's state.

__init__
__setstate__

Set the state of the object.

forward

Transform the input data.

get_applied_transform_components_factory

Create a function that returns the elements of the transform components ('mean' and 'std') applied during the most recent

get_transformed_output_factory

Create a function that returns the transformed output from the most recent forward pass.

initial_seed

Return the initial seed of the CPU device's random number generator.

manual_seed

Seed each of the random number generators.

reset_parameters

Reinitialize parameters and buffers.

seed

Seed each of the random number generators using a non-deterministic random number.

set_input_scale

Set the power of two that the input is divided by before it is transformed, and the output multiplied by afterwards.

tensor_parallel

Tensor parallelize the model across the given device mesh.

Attributes:

Name Type Description
num_diffusion_steps int

The number of steps to use in generating the transformation.

stopping_time float

The final time of the diffusion.

num_diffusion_steps property

num_diffusion_steps: int

The number of steps to use in generating the transformation.

stopping_time property

stopping_time: float

The final time of the diffusion.

__call__

__call__(
    input: Tensor,
    noise_mask: Tensor | None = None,
    **kwargs: Any,
) -> torch.Tensor

Transform the input data.

Parameters:

Name Type Description Default

input

Tensor

The input to transform.

required

noise_mask

Tensor | None

An optional mask that selects the elements of input to transform. Where the mask is False, the original input value is returned. Also used to select the elements of the sampled standard deviations to use to mask the input. If None, the entire input is transformed.

None

**kwargs

Any

Additional keyword arguments to the estimator modules.

required

Changed in version v5.1.0: The input is now divided by `input_scale` before it is transformed and the output multiplied by it afterwards. Behavior is unchanged for an input scale of `1`, which is the default.

__getstate__

__getstate__() -> dict[str, Any]

Prepare a JSON-serializable copy of the noise layer's state.

Returns:

Type Description
dict[str, Any]

A dictionary containing the configuration of the noise layer, including its type string, the state dict, and the generator

dict[str, Any]

states if they exist.

__init__

__init__(
    *args: Any,
    num_diffusion_steps: int = 11,
    stopping_time: float = 1.0,
    post_processor: PostProcessor | None = None,
    **kwargs: Any,
) -> None

Changed in version v3.78.0: Added the optional `post_processor` argument, which applies a learned transformation to the diffused output.

__setstate__

__setstate__(
    state: dict[str, Any],
    trust_remote_code: bool = False,
    third_party_model_path: str
    | PathLike[str]
    | None = None,
) -> None

Set the state of the object.

state_dict and _generators are both optional keys, and will be restored if they exist in the state.

Parameters:

Name Type Description Default

state

dict[str, Any]

The state to set.

required

trust_remote_code

bool

Whether to trust remote code when loading from HuggingFace Hub.

False

third_party_model_path

str | PathLike[str] | None

The path or huggingface reference to a third-party model to load. This is useful when loading SGTs whose internal structure depends on transformers which are not importable directly through transformers, but are present on the Hugging Face Hub.

None

forward

forward(
    input: Tensor,
    noise_mask: Tensor | None = None,
    **kwargs: Any,
) -> torch.Tensor

Transform the input data.

Parameters:

Name Type Description Default

input

Tensor

The input to transform.

required

noise_mask

Tensor | None

An optional mask that selects the elements of input to transform. Where the mask is 0, the original input value is returned. Also used to select the elements of the sampled standard deviations to use to mask the input. If None, the entire input is transformed.

None

**kwargs

Any

Additional keyword arguments to the estimator modules.

required

Returns:

Type Description
torch.Tensor

The transformed input data.

Raises:

Type Description
ValueError

If noise_mask is None.

Changed in version v3.78.0: An attached `post_processor` is now applied to the diffused output. Behavior is unchanged without one, which is the default.

get_applied_transform_components_factory

get_applied_transform_components_factory() -> Callable[
    [], dict[str, torch.Tensor]
]

Create a function that returns the elements of the transform components ('mean' and 'std') applied during the most recent forward pass.

Specifically, the applied elements are those selected by the noise mask (if supplied) and standard deviation mask (if std_estimator.masker is not None). If no masks are used, all elements are returned.

The applied transform components are returned flattened. They are returned at the scale they are applied at: the estimates multiplied by input_scale.

This function is intended to be used to log histograms of the transform components.

Returns:

Type Description
Callable[[], dict[str, torch.Tensor]]

A function that returns the the elements of the transform components applied during the most recent forward pass.

Examples:

>>> from torch import nn
>>> from stainedglass_core import model as sg_model, noise_layer as sg_noise_layer
>>> base_model = nn.Linear(20, 2)
>>> noisy_model = sg_model.NoisyModel(
...     sg_noise_layer.CloakNoiseLayer1,
...     base_model,
...     target_parameter="input",
... )
>>> get_applied_transform_components = (
...     noisy_model.noise_layer.get_applied_transform_components_factory()
... )
>>> input = torch.ones(1, 20)
>>> noise_mask = torch.tensor(5 * [False] + 15 * [True])
>>> output = noisy_model(input, noise_mask=noise_mask)
>>> applied_transform_components = get_applied_transform_components()
>>> applied_transform_components
{'mean': tensor(...), 'std': tensor(...)}
>>> {
...     component_name: component.shape
...     for component_name, component in applied_transform_components.items()
... }
{'mean': torch.Size([15]), 'std': torch.Size([15])}

Changed in version v5.1.0: The transform components are now returned at the scale they are applied at, multiplied by `input_scale`. Behavior is unchanged for an input scale of `1`, which is the default.

get_transformed_output_factory

get_transformed_output_factory() -> Callable[
    [], torch.Tensor
]

Create a function that returns the transformed output from the most recent forward pass.

If super batching is active, only the transformed half of the super batch output is returned. The output is returned at the scale of the noise layer's input, so it is unaffected by input_scale.

Returns:

Type Description
Callable[[], torch.Tensor]

A function that returns the transformed output from the most recent forward pass.

Examples:

>>> from stainedglass_core import noise_layer as sg_noise_layer
>>> noise_layer = sg_noise_layer.CloakNoiseLayer1()
>>> get_transformed_output = noise_layer.get_transformed_output_factory()
>>> input = torch.ones(2, 3, 32, 32)
>>> output = noise_layer(input)
>>> transformed_output = get_transformed_output()
>>> assert output.equal(transformed_output)

Changed in version v5.1.0: The transformed output is now returned at the scale of the noise layer's input, multiplied by `input_scale`. Behavior is unchanged for an input scale of `1`, which is the default.

initial_seed

initial_seed() -> int

Return the initial seed of the CPU device's random number generator.

manual_seed

manual_seed(
    seed: int | None, rank_dependent: bool = True
) -> None

Seed each of the random number generators.

Setting seed to None will destroy any existing generators, disabling reproducible behavior until the next call with an integer seed. Calling manual_seed again with an integer after a None call fully recreates generators for every currently available device type (CPU, CUDA, MPS) before applying the seed, making the operation safely reversible.

Parameters:

Name Type Description Default

seed

int | None

The seed to set. Pass None to disable reproducible behavior.

required

rank_dependent

bool

Whether to add the distributed rank to the seed to ensure that each process samples different noise.

True

reset_parameters

reset_parameters() -> None

Reinitialize parameters and buffers.

This method is useful for initializing tensors created on the meta device.

seed

seed() -> None

Seed each of the random number generators using a non-deterministic random number.

set_input_scale

set_input_scale(scale: float) -> None

Set the power of two that the input is divided by before it is transformed, and the output multiplied by afterwards.

With an input scale s, the noise layer returns s * T(input / s), where T is the transformation it would apply with an input scale of 1. Elements the noise mask leaves untransformed are returned exactly as they were input. The estimators, the post-processor, and every loss computed from activations captured inside the noise layer (for example its estimated means and standard deviations, or its output captured by a hook_loss_wrapper hook) see the scaled-down values. Losses computed from activations outside the noise layer, such as a base model's logits, see the noise layer's full-scale output.

The scale must be a power of two from 2**-14 to 2**14. The scale and its reciprocal are then exact in every floating point dtype, including torch.float16, so casting the noise layer to another dtype preserves the scale, and dividing and multiplying by it is exact wherever the result is a normal number.

Parameters:

Name Type Description Default

scale

float

The input scale, a power of two from 2**-14 to 2**14.

required

Examples:

A noise layer whose initialization suits inputs with a root mean square of about 0.01 can transform inputs with a root mean square of about 0.5 at the scale it was calibrated for:

>>> from stainedglass_core import noise_layer as sg_noise_layer
>>> noise_layer = sg_noise_layer.CloakNoiseLayer1(seed=0)
>>> noise_layer.set_input_scale(64.0)
>>> noise_layer.input_scale
tensor(64.)
>>> output = noise_layer(torch.randn(2, 8) * 0.5)

The scale must be a power of two:

>>> noise_layer.set_input_scale(50.0)
Traceback (most recent call last):
...
ValueError: The input scale must be a power of two from 2**-14 to 2**14, but got 50.0.

Added in version v5.1.0.

tensor_parallel

tensor_parallel(mesh: DeviceMesh) -> None

Tensor parallelize the model across the given device mesh.

Parameters:

Name Type Description Default

mesh

DeviceMesh

The tensor parallel device mesh.

required