Skip to content

huggingface

A module for utilities for Hugging Face transformers.

Functions:

Name Description
get_base_model_backbone

Get the base model decoder from a Hugging Face model.

get_bias_free_lm_head

Return the model's lm_head, checking that it projects hidden states with its weight alone.

get_base_model_backbone

get_base_model_backbone(model: PreTrainedModel) -> <class 'transformers.modeling_utils.PreTrainedModel'>

Get the base model decoder from a Hugging Face model.

Typically, Hugging Face *ForCausalLM models wrap a base model that contains the actual architecture. This model can be accessed via the get_decoder method, but some models, like Llama4ForCausalLM do not implement this method correctly in Transformers v5.0.0. This utility function attempts to retrieve the base model decoder in a more robust way.

Parameters:

Name Type Description Default

model

PreTrainedModel

The Hugging Face model.

required

Returns:

Type Description
<class 'transformers.modeling_utils.PreTrainedModel'>

The base model decoder.

Raises:

Type Description
AttributeError

If the base model decoder cannot be found.

Added in version v3.28.0. Renamed from `get_base_model_decoder` to support both encoder and decoder architectures.

get_bias_free_lm_head

get_bias_free_lm_head(
    model: Module,
) -> (
    torch.nn.modules.linear.Linear
    | stainedglass_core.utils.torch.nn.TiedLinear
)

Return the model's lm_head, checking that it projects hidden states with its weight alone.

The lm_head is the module returned by the model's get_output_embeddings().

Parameters:

Name Type Description Default

model

Module

The model whose lm_head to return.

required

Returns:

Type Description
torch.nn.modules.linear.Linear | stainedglass_core.utils.torch.nn.TiedLinear

The model's lm_head.

Raises:

Type Description
ValueError

If the model has no output embeddings.

TypeError

If the lm_head is neither a torch.nn.Linear nor a TiedLinear, e.g. because an adapter wraps it.

ValueError

If the lm_head has a bias.

Added in version v4.7.0.