projection

Functions:

Name	Description
calculate_equivalent_alpha_and_scaling_factor	Calculate the equivalent alpha and scaling factor (compared to the composite loss) for an alphaless gradient.
order_tensors_by_norm	Return the supplied tensors as a tuple, sorted by their norm.
project_out_smaller_tensor	Calculate the larger of the two supplied gradients, and then returns the larger gradient with the component of the smaller gradient
vector_projection	Compute the (vector) projection of tensor_a onto tensor_b.

calculate_equivalent_alpha_and_scaling_factor

calculate_equivalent_alpha_and_scaling_factor(
    noise_loss_grad: Tensor, model_loss_grad: Tensor
) -> tuple[torch.Tensor, torch.Tensor]

Calculate the equivalent alpha and scaling factor (compared to the composite loss) for an alphaless gradient.

This gives us a closed form solution for what alpha would be within the alphaless framework, which could be handy in trying to understand noise layer training.

Parameters:

Name	Type	Description	Default
noise_loss_grad	Tensor	The gradient of the noise loss, with respect to the trained parameters.	required
model_loss_grad	Tensor	The gradient of the model loss, with respect to the trained parameters.	required

Returns:

Type	Description
tuple[torch.Tensor, torch.Tensor]	A tuple of the equivalent alpha and scaling factor.

order_tensors_by_norm

order_tensors_by_norm(
    tensor_a: Tensor, tensor_b: Tensor, **kwargs: Any
) -> tuple[torch.Tensor, torch.Tensor]

Return the supplied tensors as a tuple, sorted by their norm.

By default, the vector 2-norm is used, however by supplying kwargs, other norms are possible to use.

Note

If the norms are equal, the order of the supplied tensors is preserved. I.e. the function is idempotent/the sort is stable.

Parameters:

Name	Type	Description	Default
tensor_a	Tensor	A tensor to compare the norm of.	required
tensor_b	Tensor	A tensor to compare the norm of.	required
**kwargs	Any	The variable length keyword arguments to supply to the underlying torch.linalg.vector_norm call. Useful for using other norms than the default vector 2-norm.	required

Returns:

Type	Description
tuple[torch.Tensor, torch.Tensor]	A tuple of the original tensors, where the first element has norm less than or equal to the second element.

project_out_smaller_tensor

project_out_smaller_tensor(tensor_a: Tensor, tensor_b: Tensor) -> <class 'torch.Tensor'>

Calculate the larger of the two supplied gradients, and then returns the larger gradient with the component of the smaller gradient removed.

This function is symmetric.

Parameters:

Name	Type	Description	Default
tensor_a	Tensor	A tensor to compare and project.	required
tensor_b	Tensor	A tensor to compare and project.	required

Returns:

Type	Description
<class 'torch.Tensor'>	The larger of the supplied tensors with the component of the smaller tensor removed.

vector_projection

vector_projection(tensor_a: Tensor, tensor_b: Tensor) -> <class 'torch.Tensor'>

Compute the (vector) projection of tensor_a onto tensor_b.

\(\text{proj}_{b}a := a\cdot\hat{b}\hat{b}\)

Parameters:

Name	Type	Description	Default
tensor_a	Tensor	The tensor to project.	required
tensor_b	Tensor	The tensor to project onto.	required

Returns:

Type	Description
<class 'torch.Tensor'>	tensor_a projected onto tensor_b.