parameterizations
Bases: ScaledStandardDeviationParameterization
Parameterizes rhos tensors (on the domain of all real numbers) as standard deviations tensors (on the open domain of min_scale to
max_scale).
min_scale is strictly less than max_scale, and both must be nonnegative real numbers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
tuple[float, float]
|
The asymptotic minimum and maximum values of the parameterized standard deviations. |
(0.0001, 2.0)
|
|
float
|
A temperature-like parameter which controls the spread of the parameterization. Controls both the magnitude of
parameterized standard deviations and their rate of change with respect to rhos. |
1.0
|
Added in version 0.11.0.
Reinitialize parameters and buffers.
This method is useful for initializing tensors created on the meta device.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
Bases: ScaledStandardDeviationParameterization
A direct parameterization of rhos tensors as standard deviations tensors (clamped into the closed domain of min_scale to
max_scale).
Reinitialize parameters and buffers.
This method is useful for initializing tensors created on the meta device.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
Bases: StandardDeviationParameterization
Defines the common structures necessary to parameterize rhos tensors (on the domain of all real numbers) as standard deviation
tensors (on the domain of min_scale to max_scale).
min_scale is strictly less than max_scale, and both must be nonnegative real numbers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
tuple[float, float]
|
The minimum and maximum values of the parameterized standard deviations. |
(0.0001, 2.0)
|
abstractmethod
¶
abstractmethod
¶
Reinitialize parameters and buffers.
This method is useful for initializing tensors created on the meta device.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
ValueError
|
If |
Defines the interface for the reparameterization of rhos tensors (on the domain of all real numbers) as standard deviation tensors (on the domain of nonnegative real numbers) of the applied transformation.
Rhos can be learned directly or estimated as the output of a neural network.
The derivative of this parameterization defines the rate of change of the standard deviations with respect rhos. When combined with a so-called "noise loss", used to penalize the standard deviations from deviating from the target distribution, and a task loss (i.e. classification, token prediction), we define a complete training objective and loss landscape for transform layers.
Added in version 0.11.0.