Skip to content

prompt_embeds_loader

vLLM's prompt-embeds loader, patched to decode the prompt_embeds wire formats this package's plugins register decoders for.

vLLM loads each prompt_embeds payload with safe_load_prompt_embeds, which only accepts a tensor. The client (e.g SGT Proxy) may instead send a torch.save-ed dict of another wire format, such as TurboQuant-compressed embeddings or SVD coefficients, each identified by a key that only its dicts carry. Each format is decoded by its own vLLM general plugin, which registers a decoder for its key with register_decoder. The first registration patches vLLM's loader, so the plugins neither know about one another nor depend on the order vLLM registers them.

The patched loader deserializes each payload. It loads a plain tensor as vLLM does, and hands a dict to the decoder registered for the key it carries, together with a DecodeTarget saying what the model takes. Whatever the format, it then checks the result against the model as vLLM's loader does. A decoder whose payload nests a payload of another format decodes it with decode and a target of its own, which is how formats compose: SVD coefficients may be TurboQuant-compressed, and are then decoded by the TurboQuant plugin. A dict of a format no loaded plugin decodes is refused with a validation error, as vLLM's own loader refuses any dict.

Warning

Under almost no circumstances should you need to import this module directly. The plugins that decode prompt_embeds use it.

Type Aliases:

Name Description
PayloadDecoder

Decodes a payload dict of one wire format, as deserialized, to a tensor of shape (num_tokens, dim) on the CPU in the target's dtype.

Classes:

Name Description
DecodeTarget

What the caller of a decoder takes, which the decoder checks a payload against before it spends any memory on decoding it.

Functions:

Name Description
decode

Decode a payload dict with the decoder registered for its wire format.

decode_device

Return the device this process decodes prompt_embeds payloads on: the active CUDA device, or the CPU if CUDA is unavailable.

register_decoder

Have vLLM's prompt-embeds loader decode the payload dicts that carry key with decoder.

PayloadDecoder

PayloadDecoder = Callable[
    [Mapping[str, object], DecodeTarget], Tensor
]

Decodes a payload dict of one wire format, as deserialized, to a tensor of shape (num_tokens, dim) on the CPU in the target's dtype.

It runs the target's checks before it spends any memory on decoding, and raises VLLMValidationError for a payload it cannot decode.

DecodeTarget dataclass

What the caller of a decoder takes, which the decoder checks a payload against before it spends any memory on decoding it.

The loader decodes for the model, so its target takes embeddings of the model's hidden size, in the model's dtype, for at most the model's context limit of tokens, and at most VLLM_MAX_EMBED_DECODE_BYTES of them all together. A decoder that decodes a payload nested in its own passes a target of its own, which keeps both limits: every token of the nested payload still decodes to one embedding for the model.

Methods:

Name Description
check_num_tokens

Check that a payload of num_tokens tokens fits in the model's context limit, and decodes within VLLM_MAX_EMBED_DECODE_BYTES.

Attributes:

Name Type Description
bytes_per_token int | None

The bytes every token a payload carries takes once decoded for the model, which count against VLLM_MAX_EMBED_DECODE_BYTES. None

check_dim Callable[[int], None]

Checks the number of values per token a payload decodes to, raising VLLMValidationError if the caller cannot take it.

dtype dtype

The dtype to decode into.

max_model_len int | None

The model's context limit, which every token a payload carries counts against. None skips the check.

bytes_per_token instance-attribute

bytes_per_token: int | None

The bytes every token a payload carries takes once decoded for the model, which count against VLLM_MAX_EMBED_DECODE_BYTES. None skips the check.

check_dim instance-attribute

check_dim: Callable[[int], None]

Checks the number of values per token a payload decodes to, raising VLLMValidationError if the caller cannot take it.

dtype instance-attribute

dtype: dtype

The dtype to decode into.

max_model_len instance-attribute

max_model_len: int | None

The model's context limit, which every token a payload carries counts against. None skips the check.

check_num_tokens

check_num_tokens(num_tokens: int) -> None

Check that a payload of num_tokens tokens fits in the model's context limit, and decodes within VLLM_MAX_EMBED_DECODE_BYTES.

vLLM bounds the memory a prompt_embeds tensor may allocate once densified by VLLM_MAX_EMBED_DECODE_BYTES, but only ever sees the dicts the plugins decode as they are sent, so the bound is checked here, against what they decode to. That can be far larger than the payload: SVD coefficients decode to hidden_size values per token from as few as one, so a payload of a few hundred KiB within the context limit decodes to GiBs.

Parameters:

Name Type Description Default

num_tokens

int

The number of tokens the payload carries.

required

Raises:

Type Description
VLLMValidationError

If num_tokens exceeds max_model_len, or their embeddings would exceed VLLM_MAX_EMBED_DECODE_BYTES.

decode

decode(
    payload: Mapping[str, object], target: DecodeTarget
) -> torch.Tensor

Decode a payload dict with the decoder registered for its wire format.

Parameters:

Name Type Description Default

payload

Mapping[str, object]

The payload dict, as deserialized.

required

target

DecodeTarget

What the caller takes.

required

Returns:

Type Description
torch.Tensor

The decoded tensor, of shape (num_tokens, dim) for a dim that target.check_dim accepts, on the CPU in target.dtype.

Raises:

Type Description
VLLMValidationError

If the payload carries the key of no registered wire format, or of more than one, or its decoder refuses it.

decode_device cached

decode_device() -> torch.device

Return the device this process decodes prompt_embeds payloads on: the active CUDA device, or the CPU if CUDA is unavailable.

Cached per process: the device assigned to this process by vLLM never changes after startup, so there is no need to re-evaluate on every request.

Returns:

Type Description
torch.device

The device to decode on.

register_decoder

register_decoder(key: str, decoder: PayloadDecoder) -> None

Have vLLM's prompt-embeds loader decode the payload dicts that carry key with decoder.

The first registration in a process patches vLLM's loader. Registering the decoder a key already has does nothing, so that the plugins' registrations stay idempotent.

Parameters:

Name Type Description Default

key

str

The key that only the dicts of the decoder's wire format carry.

required

decoder

PayloadDecoder

The decoder.

required

Raises:

Type Description
ValueError

If another decoder is already registered for key.