middleware

Middleware for the vLLM OpenAI-compatible RESTful API server that reads a user-provided public key from the request headers, registers it with a local key registry for the duration of the request, adds the server's public key to the response headers, and then finally clears the client key from the registry after the response is sent.

Warning

Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, and you launch vLLM via the alternative entrypoint, this module will be automatically applied.

Classes:

Name	Description
UserPublicKeyMiddleware	Middleware that reads a user-provided public key from a completions endpoint POST request, and use it to generate a shared secret.

Functions:

Name	Description
inject_middleware	Add the UserPublicKeyMiddleware to a function that builds a vLLM OpenAI-compatible FastAPI application.

UserPublicKeyMiddleware

Bases: BaseHTTPMiddleware

Middleware that reads a user-provided public key from a completions endpoint POST request, and use it to generate a shared secret.

Methods:

Name	Description
__init__	Initialize the middleware and look up the client public key header from environment variable.
dispatch	Intercept the user-provided public key from headers, generate the shared secret, then register that shared key.

Attributes:

Name	Type	Description
key_registry	MutableMapping[str, bytes | None]	Reference to the class's shared key registry.

key_registry property

key_registry: MutableMapping[str, bytes | None]

Reference to the class's shared key registry.

UserPublicKeyMiddleware uses the class-level _RegistryFactory (_key_registry_factory) to maintain a single shared registry per process, so that all UserPublicKeyMiddleware instances share the same underlying registry.

__init__

__init__(
    app: ASGIApp, dispatch: DispatchFunction | None = None
) -> None

Initialize the middleware and look up the client public key header from environment variable.

Parameters:

Name	Type	Description	Default
app	ASGIApp	The FastAPI application instance.	required
dispatch	DispatchFunction | None	Optional dispatch function for handling requests. If None, uses the default dispatch behavior.	None

dispatch async

dispatch(
    request: Request, call_next: RequestResponseEndpoint
) -> fastapi.Response

Intercept the user-provided public key from headers, generate the shared secret, then register that shared key.

Note

The user must provide the public key using the header specified by the SG_CLIENT_PUBLIC_KEY_HEADER_NAME environment variable (defaults to x-client-public-key). This must be a X25519 public key, base64 encoded.

Note

The response will have a x-server-public-key header, or whatever you set SG_SERVER_PUBLIC_KEY_HEADER_NAME environment variable to, which contains the public key that the Inference Server used to encrypt the response. The client can use this public key and its private key to derive a shared secret to decrypt the text in the response. This public key is an X25519 public key, base64 encoded.

Parameters:

Name	Type	Description	Default
request	Request	A FastAPI request.	required
call_next	RequestResponseEndpoint	Function returning a coroutine which calls the endpoint.	required

Returns:

Type	Description
fastapi.Response	A Completions Response (with additional x-server-public-key header).

Raises:

Type	Description
ValueError	If the user did not include a x-client-public-key header item.
TypeError	If the server's ephemeral keys are not properly instantiated.

inject_middleware

inject_middleware(
    build_app_func: Callable[[Namespace], FastAPI],
) -> Callable[[argparse.Namespace], fastapi.FastAPI]

Add the UserPublicKeyMiddleware to a function that builds a vLLM OpenAI-compatible FastAPI application.

Parameters:

Name	Type	Description	Default
build_app_func	Callable[[Namespace], FastAPI]	Function that builds the vLLM OpenAI-compatible FastAPI application	required

Returns:

Type	Description
Callable[[argparse.Namespace], fastapi.FastAPI]	A new function compatible with the same signature as build_app_func that also adds the UserPublicKeyMiddleware after using
Callable[[argparse.Namespace], fastapi.FastAPI]	build_app_func to build the app.