huggingface
A module for utilities for Hugging Face transformers.
Functions:
| Name | Description |
|---|---|
get_base_model_backbone |
Get the base model decoder from a Hugging Face model. |
get_bias_free_lm_head |
Return the model's lm_head, checking that it projects hidden states with its weight alone. |
get_base_model_backbone
¶
get_base_model_backbone(model: PreTrainedModel) -> <class 'transformers.modeling_utils.PreTrainedModel'>
Get the base model decoder from a Hugging Face model.
Typically, Hugging Face *ForCausalLM models wrap a base model that contains the actual architecture.
This model can be accessed via the get_decoder method, but some models, like Llama4ForCausalLM do not implement this method
correctly in Transformers v5.0.0. This utility function attempts to retrieve the base model decoder in a more robust way.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
PreTrainedModel
|
The Hugging Face model. |
required |
Returns:
| Type | Description |
|---|---|
<class 'transformers.modeling_utils.PreTrainedModel'>
|
The base model decoder. |
Raises:
| Type | Description |
|---|---|
AttributeError
|
If the base model decoder cannot be found. |
Added in version v3.28.0. Renamed from `get_base_model_decoder` to support both encoder and decoder architectures.
get_bias_free_lm_head
¶
get_bias_free_lm_head(
model: Module,
) -> (
torch.nn.modules.linear.Linear
| stainedglass_core.utils.torch.nn.TiedLinear
)
Return the model's lm_head, checking that it projects hidden states with its weight alone.
The lm_head is the module returned by the model's get_output_embeddings().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Module
|
The model whose lm_head to return. |
required |
Returns:
| Type | Description |
|---|---|
torch.nn.modules.linear.Linear | stainedglass_core.utils.torch.nn.TiedLinear
|
The model's lm_head. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the model has no output embeddings. |
TypeError
|
If the lm_head is neither a |
ValueError
|
If the lm_head has a bias. |
Added in version v4.7.0.