client_key_middleware

Middleware for the vLLM OpenAI-compatible RESTful API server that reads a user-provided public key from the request headers, registers it with a local key registry for the duration of the request, adds the server's public key to the response headers, and then finally clears the client key from the registry after the response is sent.

Warning

Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, vLLM loads it automatically via the vllm.general_plugins entry point.

Classes:

Name	Description
UserPublicKeyMiddleware	Middleware that reads a user-provided public key from a completions endpoint POST request, and use it to generate a shared secret.

Functions:

Name	Description
inject_middleware	Add the UserPublicKeyMiddleware to a function that builds a vLLM OpenAI-compatible FastAPI application.

Attributes:

Name	Type	Description
USER_PUBLIC_KEY_MIDDLEWARE_PATH	Final[str]	Import path that vLLM's --middleware loader uses to install UserPublicKeyMiddleware.

USER_PUBLIC_KEY_MIDDLEWARE_PATH module-attribute

USER_PUBLIC_KEY_MIDDLEWARE_PATH: Final[str] = (
    f"{UserPublicKeyMiddleware.__module__}.{UserPublicKeyMiddleware.__name__}"
)

Import path that vLLM's --middleware loader uses to install UserPublicKeyMiddleware.

UserPublicKeyMiddleware

Middleware that reads a user-provided public key from a completions endpoint POST request, and use it to generate a shared secret.

This is a pure ASGI middleware rather than a starlette.middleware.base.BaseHTTPMiddleware. BaseHTTPMiddleware routes every response body through an anyio task group and a memory object stream, which costs roughly 11us of event-loop CPU per SSE chunk — on a streaming inference server that is paid once per generated token, per request, on the single thread that also runs encryption and response serialization. Handling the ASGI messages directly costs approximately nothing, and lets non-protected paths (/health, /metrics) pass through before any of this machinery is set up.

Methods:

Name	Description
__call__	Intercept the user-provided public key from headers, generate the shared secret, then register that shared key.
__init__	Initialize the middleware and look up the client public key header from environment variable.

Attributes:

Name	Type	Description
key_registry	MutableMapping[str, SessionKey | None]	Reference to the class's shared key registry.

key_registry property

key_registry: MutableMapping[str, SessionKey | None]

Reference to the class's shared key registry.

UserPublicKeyMiddleware uses the class-level _RegistryFactory (_key_registry_factory) to maintain a single shared registry per process, so that all UserPublicKeyMiddleware instances share the same underlying registry.

__call__ async

__call__(
    scope: Scope, receive: Receive, send: Send
) -> None

Intercept the user-provided public key from headers, generate the shared secret, then register that shared key.

Note

The user must provide the public key using the header specified by the SG_CLIENT_PUBLIC_KEY_HEADER_NAME environment variable (defaults to x-client-public-key). This must be a X25519 public key, base64 encoded.

Note

The response will have a x-server-public-key header, or whatever you set SG_SERVER_PUBLIC_KEY_HEADER_NAME environment variable to, which contains the public key that the Inference Server used to encrypt the response. The client can use this public key and its private key to derive a shared secret to decrypt the text in the response. This public key is an X25519 public key, base64 encoded.

Parameters:

Name	Type	Description	Default
scope	Scope	The ASGI connection scope.	required
receive	Receive	The ASGI receive channel.	required
send	Send	The ASGI send channel.	required

Raises:

Type	Description
ValueError	If the user did not include a x-client-public-key header item.
TypeError	If the server's ephemeral keys are not properly instantiated.

__init__

__init__(app: ASGIApp) -> None

Initialize the middleware and look up the client public key header from environment variable.

Parameters:

Name	Type	Description	Default
app	ASGIApp	The next ASGI application in the stack.	required

inject_middleware

inject_middleware(
    build_app_func: BuildAppFunc,
) -> BuildAppFunc

Add the UserPublicKeyMiddleware to a function that builds a vLLM OpenAI-compatible FastAPI application.

Parameters:

Name	Type	Description	Default
build_app_func	BuildAppFunc	Function that builds the vLLM OpenAI-compatible FastAPI application	required

Returns:

Type	Description
BuildAppFunc	A new function compatible with the same signature as build_app_func that also adds the UserPublicKeyMiddleware after using
BuildAppFunc	build_app_func to build the app.