turboquant_plugin
vLLM general plugin that decodes TurboQuant-compressed prompt_embeds payloads.
Register via the vllm.general_plugins entry point (declared in pyproject.toml).
As of vLLM 0.21.0, load_general_plugins() is called in every process: in the API server via
EngineArgs.create_engine_config (vllm/engine/arg_utils.py:731), and in the engine subprocess
via EngineCore.__init__ (vllm/v1/engine/core.py:105). The plugin is therefore active in all
processes without any explicit call from entrypoint.py.
The wire format sent by the proxy is a torch.save-ed dict:
The plugin registers a decoder for the dicts that carry packed with the
prompt-embeds loader, which hands it every such dict: a whole payload of
TurboQuant-compressed embeddings, or one nested in a payload of another wire format, such as TurboQuant-compressed SVD coefficients. Each call
says what its caller takes, so this plugin knows nothing of the formats that nest its payloads.
Plain float tensors (the existing uncompressed path) are passed through unchanged, so deployment is fully backwards-compatible: old proxy → patched vLLM works, and new proxy → unpatched vLLM fails gracefully (vLLM rejects the dict payload with a clear validation error rather than a silent wrong result).
Warning
Under almost no circumstances should you need to import this module directly.
If stainedglass_output_protection is installed, vLLM will load the plugin automatically.
Functions:
| Name | Description |
|---|---|
register |
Have vLLM's prompt-embeds loader decode TurboQuant-compressed payloads, and compile the decode of the dimensions to expect. |
register
¶
Have vLLM's prompt-embeds loader decode TurboQuant-compressed payloads, and compile the decode of the dimensions to expect.
Called automatically by vLLM's plugin system at startup in all processes —
both the main OpenAI API server process (via EngineArgs.create_engine_config,
vllm/engine/arg_utils.py:731) and the engine subprocess (via EngineCore.__init__,
vllm/v1/engine/core.py:105). Safe to call multiple times (idempotent).
Registering the decoder with the prompt-embeds loader patches vLLM's loader, unless another plugin already has.