prompt_embeds_loader
vLLM's prompt-embeds loader, patched to decode the prompt_embeds wire formats this package's plugins register decoders for.
vLLM loads each prompt_embeds payload with safe_load_prompt_embeds, which only accepts a tensor. The client (e.g SGT Proxy) may instead send a
torch.save-ed dict of another wire format, such as TurboQuant-compressed embeddings or SVD coefficients, each identified by a key that only
its dicts carry. Each format is decoded by its own vLLM general plugin, which registers a decoder for its key with
register_decoder. The first registration patches vLLM's
loader, so the plugins neither know about one another nor depend on the order vLLM registers them.
The patched loader deserializes each payload. It loads a plain tensor as vLLM does, and hands a dict to the decoder registered for the key it
carries, together with a DecodeTarget saying what the model takes.
Whatever the format, it then checks the result against the model as vLLM's loader does. A decoder whose payload nests a payload of another
format decodes it with decode and a target of its own, which is how
formats compose: SVD coefficients may be TurboQuant-compressed, and are then decoded by the TurboQuant plugin. A dict of a format no loaded
plugin decodes is refused with a validation error, as vLLM's own loader refuses any dict.
Warning
Under almost no circumstances should you need to import this module directly. The plugins that decode prompt_embeds use it.
Type Aliases:
| Name | Description |
|---|---|
PayloadDecoder |
Decodes a payload dict of one wire format, as deserialized, to a tensor of shape |
Classes:
| Name | Description |
|---|---|
DecodeTarget |
What the caller of a decoder takes, which the decoder checks a payload against before it spends any memory on decoding it. |
Functions:
| Name | Description |
|---|---|
decode |
Decode a payload dict with the decoder registered for its wire format. |
decode_device |
Return the device this process decodes |
register_decoder |
Have vLLM's prompt-embeds loader decode the payload dicts that carry |
PayloadDecoder
¶
Decodes a payload dict of one wire format, as deserialized, to a tensor of shape (num_tokens, dim) on the CPU in the target's dtype.
It runs the target's checks before it spends any memory on decoding, and raises VLLMValidationError for a payload it cannot decode.
DecodeTarget
dataclass
¶
What the caller of a decoder takes, which the decoder checks a payload against before it spends any memory on decoding it.
The loader decodes for the model, so its target takes embeddings of the model's hidden size, in the model's dtype, for at most the model's
context limit of tokens, and at most VLLM_MAX_EMBED_DECODE_BYTES of them all together. A decoder that decodes a payload nested in its
own passes a target of its own, which keeps both limits: every token of the nested payload still decodes to one embedding for the model.
Methods:
| Name | Description |
|---|---|
check_num_tokens |
Check that a payload of |
Attributes:
| Name | Type | Description |
|---|---|---|
bytes_per_token |
int | None
|
The bytes every token a payload carries takes once decoded for the model, which count against |
check_dim |
Callable[[int], None]
|
Checks the number of values per token a payload decodes to, raising |
dtype |
dtype
|
The dtype to decode into. |
max_model_len |
int | None
|
The model's context limit, which every token a payload carries counts against. |
bytes_per_token
instance-attribute
¶
bytes_per_token: int | None
The bytes every token a payload carries takes once decoded for the model, which count against VLLM_MAX_EMBED_DECODE_BYTES. None
skips the check.
check_dim
instance-attribute
¶
Checks the number of values per token a payload decodes to, raising VLLMValidationError if the caller cannot take it.
max_model_len
instance-attribute
¶
max_model_len: int | None
The model's context limit, which every token a payload carries counts against. None skips the check.
check_num_tokens
¶
check_num_tokens(num_tokens: int) -> None
Check that a payload of num_tokens tokens fits in the model's context limit, and decodes within VLLM_MAX_EMBED_DECODE_BYTES.
vLLM bounds the memory a prompt_embeds tensor may allocate once densified by VLLM_MAX_EMBED_DECODE_BYTES, but only ever sees
the dicts the plugins decode as they are sent, so the bound is checked here, against what they decode to. That can be far larger
than the payload: SVD coefficients decode to hidden_size values per token from as few as one, so a payload of a few hundred KiB
within the context limit decodes to GiBs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
int
|
The number of tokens the payload carries. |
required |
Raises:
| Type | Description |
|---|---|
VLLMValidationError
|
If |
decode
¶
Decode a payload dict with the decoder registered for its wire format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
Mapping[str, object]
|
The payload dict, as deserialized. |
required |
|
DecodeTarget
|
What the caller takes. |
required |
Returns:
| Type | Description |
|---|---|
torch.Tensor
|
The decoded tensor, of shape |
Raises:
| Type | Description |
|---|---|
VLLMValidationError
|
If the payload carries the key of no registered wire format, or of more than one, or its decoder refuses it. |
decode_device
cached
¶
Return the device this process decodes prompt_embeds payloads on: the active CUDA device, or the CPU if CUDA is unavailable.
Cached per process: the device assigned to this process by vLLM never changes after startup, so there is no need to re-evaluate on every request.
Returns:
| Type | Description |
|---|---|
torch.device
|
The device to decode on. |
register_decoder
¶
register_decoder(key: str, decoder: PayloadDecoder) -> None
Have vLLM's prompt-embeds loader decode the payload dicts that carry key with decoder.
The first registration in a process patches vLLM's loader. Registering the decoder a key already has does nothing, so that the plugins' registrations stay idempotent.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
|
str
|
The key that only the dicts of the decoder's wire format carry. |
required |
|
PayloadDecoder
|
The decoder. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If another decoder is already registered for |