Skip to content

turboquant_plugin

vLLM general plugin that decodes TurboQuant-compressed prompt_embeds payloads.

Register via the vllm.general_plugins entry point (declared in pyproject.toml). As of vLLM 0.21.0, load_general_plugins() is called in every process: in the API server via EngineArgs.create_engine_config (vllm/engine/arg_utils.py:731), and in the engine subprocess via EngineCore.__init__ (vllm/v1/engine/core.py:105). The plugin is therefore active in all processes without any explicit call from entrypoint.py.

The wire format sent by the proxy is a torch.save-ed dict:

{"packed": <uint8 tensor>, "norms": <float32 tensor>, "bits": <int>, "d": <int>}

The plugin registers a decoder for the dicts that carry packed with the prompt-embeds loader, which hands it every such dict: a whole payload of TurboQuant-compressed embeddings, or one nested in a payload of another wire format, such as TurboQuant-compressed SVD coefficients. Each call says what its caller takes, so this plugin knows nothing of the formats that nest its payloads.

Plain float tensors (the existing uncompressed path) are passed through unchanged, so deployment is fully backwards-compatible: old proxy → patched vLLM works, and new proxy → unpatched vLLM fails gracefully (vLLM rejects the dict payload with a clear validation error rather than a silent wrong result).

Warning

Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, vLLM will load the plugin automatically.

Functions:

Name Description
register

Have vLLM's prompt-embeds loader decode TurboQuant-compressed payloads, and compile the decode of the dimensions to expect.

register

register() -> None

Have vLLM's prompt-embeds loader decode TurboQuant-compressed payloads, and compile the decode of the dimensions to expect.

Called automatically by vLLM's plugin system at startup in all processes — both the main OpenAI API server process (via EngineArgs.create_engine_config, vllm/engine/arg_utils.py:731) and the engine subprocess (via EngineCore.__init__, vllm/v1/engine/core.py:105). Safe to call multiple times (idempotent).

Registering the decoder with the prompt-embeds loader patches vLLM's loader, unless another plugin already has.