Stained Glass Output Protection¶
Stained Glass Output Protection is a library and associated vLLM plugin for tokenwise encrypting messages generated by a large language model.
Deployment¶
Docker¶
The Stained Glass Output Protection docker image is built from the official vLLM image and includes the installed plugin, which vLLM loads automatically. It uses the same vllm serve entrypoint and exposes the same ports as the vLLM image, and enables Prompt Embeds support by default.
Once the provided docker image is acquired, it can be run with the following command:
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--env "SG_CLIENT_PUBLIC_KEY_HEADER_NAME=x-client-public-key" \ # pragma: allowlist secret
-p 8000:8000 \
--ipc=host \
protopia-ai/stainedglass-inference-server:0.5.1-e6205f3 <or the tag you have> \
--model meta-llama/Meta-Llama-3.1-8B-Instruct
The resulting vLLM server can be available at http://localhost:8000/, and will expose an OpenAI compatible API that accepts prompt embeds.
Any CLI arguments that are valid for vLLM can be passed to the container in the docker run command.
You can also mount any volumes containing model weights. The above mounts the local user's Hugging Face cache directory to the container's Hugging Face cache directory, which is useful for models that are downloaded from Hugging Face.
ARM64 Support
The Stained Glass Inference Server docker image (based on the vLLM docker image with Stained Glass Output Protection pre-installed and Prompt Embeddings support automatically enabled) is also available for ARM64 architectures (such as for DGX Spark). It should the same system compatibility as the official vLLM docker image (i.e. no Apple Silicon support). The arm64 image is considered experimental. Please contact Protopia AI support for access.
Docker Compose¶
Alternatively, you can launch the Stained Glass Output Protection docker image using Docker Compose. The following example docker-compose.yml file can be used:
---
services:
model-server:
image: stainedglass-inference-server:${TAG:-latest} # Specify the TAG via an environment variable, or manually set it
ports:
- "8000:8000"
volumes:
- "~/.cache/huggingface:/root/.cache/huggingface"
command: --model meta-llama/Meta-Llama-3.1-8B-Instruct
environment:
- HF_TOKEN=<secret>
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['0']
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 1m30s
timeout: 30s
retries: 5
start_period: 30s
Python Wheel (vLLM Server)¶
Stained Glass Output Protection is also available as a Python wheel, which can be installed in any Python (>=3.12) environment via pip or uv.
# pip
VLLM_USE_PRECOMPILED=1 pip install "stainedglass_output_protection-1.1.1-py312-none-any.whl[vllm]"
# uv
VLLM_USE_PRECOMPILED=1 uv install "stainedglass_output_protection-1.1.1-py312-none-any.whl[vllm]"
In either case, the vllm extra will also install the associated version of the vLLM library. Using the VLLM_USE_PRECOMPILED=1 environment variable ensures that a pre-compiled vLLM wheel is used to reduce installation time, but this is technically optional. The wheel filename may vary based on the Python version and platform, so you may need to adjust the filename accordingly.
This installs the vLLM plugin, which vLLM loads automatically in every one of its processes when it starts. No special entrypoint is needed, so launch vLLM the usual way (Prompt embeddings are enabled by default):
The resulting vLLM server can be available at http://localhost:8000/, and will expose an OpenAI compatible API that accepts prompt embeds.
Any CLI arguments that are valid for vllm serve can be passed to the container in this command.
When launched this way, vLLM will automatically use all the available GPUs on the system. You can use the CUDA_VISIBLE_DEVICES environment variable to limit the GPUs that vLLM uses.
Deprecated
Earlier versions required launching via python -m stainedglass_output_protection.vllm.entrypoint. That still works, but is deprecated and now simply delegates to vllm serve.
Debugging without client keys¶
Plaintext responses: debugging only
SG_DEBUG_ENABLE_REQUESTS_WITH_NO_KEYS defaults to false. Set it to true (or 1) before starting the server to accept
generation requests without a client public key. The server logs a startup warning and a warning containing the request ID for each
keyless request. Do not enable this in production.
Keyless requests use vLLM's normal generation path: no output encryption or token-ID scrubbing, normal log probabilities when requested, normal special-token handling, and server-side tool/reasoning parsing and reasoning-token counting when configured in vLLM. Both JSON and streaming SSE responses work with ordinary OpenAI clients without custom decryption. These responses leave the server as plaintext and have no server-public-key header. Tool and reasoning support still requires the appropriate vLLM parser configuration.
Requests with a client public key still use the normal key exchange and return encrypted responses. Malformed keys are not treated as
missing keys. Set SG_DEBUG_ENABLE_REQUESTS_WITH_NO_KEYS=false (or 0), or unset it, and restart the server to require keys again.
Other flag values cause startup to fail.
The debug mode covers /v1/completions, /v1/chat/completions, and /invocations. It does not bypass other authentication.
Python Wheel (Client)¶
Stained Glass Output Protection is also available as a Python wheel, which can be installed in any Python (>=3.12) environment via pip or uv. Unlike on the server, clients do not need to install vLLM. The client library contains utilities for generating client keys, and decrypting responses from the server.
Two client features are built on vLLM itself, and both come from the optional vllm-client extra: splitting decrypted output into
reasoning, content, and tool calls, and naming prompt_embeds payloads by a content hash so they can be uploaded once and referenced by
"id" thereafter. Installing the extra pulls vLLM, so clients that need neither feature should leave it out.
Use stainedglass_output_protection.prompt_embeds_id.compute_prompt_embeds_id to identify a prompt_embeds payload. Compute it over
the payload as sent — the object handed to torch.save — so that the Inference Server arrives at the same identifier from the object
torch.load returns.
Use stainedglass_output_protection.parsing.StreamingParser for streaming responses or
stainedglass_output_protection.parsing.parse_message for complete responses after decrypting their content. Create a new
StreamingParser for each request because it retains parser state independently for every response choice. The parser name and tokenizer
must match the model that generated the output.
This client-side parser is separate from the Output Protection vLLM plugin. The server plugin deliberately passes encrypted output through without parsing it; only a client holding the decryption key can parse the resulting plaintext.
Usage¶
Using vLLM with Stained Glass Output Protection occurs in three phases: Generating client keys, submitting requests, and decrypting responses.
Generating Client Keys¶
The Output Protection plugin requires clients to send an x25517 public key in the request headers that has been base64 encoded. Clients must provide this themselves, and the stainedglass_output_protection library provides a utilities for this.
from stainedglass_output_protection import encryption
client_private_key, client_public_key = encryption.generate_ephemeral_keypair()
headers = {
"x-client-public-key": base64.b64encode(
client_public_key.public_bytes_raw()
).decode("utf-8")
}
Submitting Requests¶
The Output Protection plugin is fully compatible with the OpenAI Python SDK. Instead of using client.completions.create, or client.chat.completions.create, you should use client.completions.with_raw_response.create or client.chat.completions.with_raw_response.create respectively. The server will return its public key (for decryption) in the response headers. Here's how to do it for /v1/completions:
openai_client = openai.AsyncOpenAI(
api_key="<API_KEY>",
base_url="<vllm server url>/v1",
default_headers=headers,
)
response = await openai_client.completions.with_raw_response.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
prompt="Please tell me about the history of Rome.",
stream_options=stream_options_type.ChatCompletionStreamOptionsParam(
include_usage=True
),
)
Note
If not using the OpenAI Python SDK, make sure that the x-client-public-key header is included in the request.
Decrypting Responses¶
Decrypting responses requires first deriving a shared AES key using the client's private key and the server's public key, which is provided in the response headers. The stainedglass_output_protection library provides utilities for this as well.
After a shared key is derived, the text of the responses can be decrypted using the decrypt_str function. Here's how to do it for /v1/completions:
server_public_key = x25519.X25519PublicKey.from_public_bytes(
base64.b64decode(response.headers["x-server-public-key"])
)
shared_key = encryption.derive_shared_aes_key(
client_private_key, server_public_key
)
completion = response.parse()
for choice in completion.choices:
print(encryption.decrypt_str(choice.text, shared_aes_key=shared_key))
Note
If not using the OpenAI Python SDK, make sure to parse the corresponding response header (x-server-public-key) and use it to derive the shared AES key, as shown above.
Configuration¶
The Output Protection plugin can be configured via environment variables.
| Environment Variable | Description |
|---|---|
HF_TOKEN |
The Hugging Face Hub token used to authenticate with the Hugging Face Hub. This is required for downloading models from the Hugging Face Hub. This is a requirement of vLLM to download gated models from the Huggingface Hub. |
SG_ENABLE_PROMPT_EMBEDS |
Set to 0 to stop defaulting vLLM's --enable-prompt-embeds to True, restoring vLLM's own default. Prompt embeddings are enabled by default so that a server started by a command you do not control still accepts Stained Glass Transformed prompts. Opt-outs remain available: --no-enable-prompt-embeds |
SG_PROMPT_EMBEDS_CACHE_GIB |
Size in gibibytes of the prompt-embeds cache, which lets a multi-turn client reference embeddings it has already uploaded instead of re-sending them each turn. Fractions are allowed, so 0.5 is half a gibibyte. Defaults to 0, which disables the cache; a value that is not a non-negative number is refused at startup rather than silently disabling it. Size it against host memory, not GPU memory. See Prompt-embeds caching below. |
SG_ENABLE_SHM_TENSOR_ARENA |
Set to 0 to leave vLLM's shared-memory tensor arena exactly as vLLM ships it: disabled by default and at vLLM's sizing. By default the arena is enabled, see Shared-memory tensor arena below. --no-enable-shm-tensor-arena also turns it off. |
SG_SHM_TENSOR_ARENA_SLOT_GIB |
Size in gibibytes of one arena slot, the largest prompt-embeddings tensor that takes the fast path. Fractions are allowed. Unset, vLLM's 256 MiB is kept. A value that is not a positive number is refused at startup. |
SG_SHM_TENSOR_ARENA_SLOTS |
Number of arena slots, which bounds how many large prompts can be sent through the arena at once. Unset, vLLM's 8 is kept. A value that is not a positive integer is refused at startup. |
SG_PRIVACY_ENCODER_SVD_PATH |
Path to the SVD of the privacy encoder's output, which the server needs to accept prompt_embeds sent as SVD coefficients. Unset, those requests are refused and every other request is served as usual. A file that cannot be loaded or does not hold a valid SVD is refused at startup, and requests whose coefficients were computed with another SVD are refused. Leave the file in place while the server runs: the first such request reads it again. See Prompt embeddings sent as SVD coefficients below. |
SG_TURBOQUANT_WARMUP_DIMS |
Sizes to compile the TurboQuant decode for at startup, comma-separated, e.g. 4096,256: the model's hidden size if the proxy sends TurboQuant-compressed embeddings, and K if it sends TurboQuant-compressed SVD coefficients. A size left out is compiled on its first request instead, which takes seconds and holds up every other prompt_embeds request. An entry that is not a power of 2 is skipped with a warning. |
VLLM_MAX_EMBED_DECODE_BYTES |
vLLM's limit, in bytes, on the memory one prompt_embeds payload may take once decoded. Defaults to 2147483648 (2 GiB); 0 disables it. vLLM applies it to tensors as they are sent. The server also applies it to TurboQuant-compressed embeddings and to SVD coefficients, by the size of the embeddings they decode to: num_tokens × hidden_size values in the model's dtype, which for SVD coefficients can be thousands of times the payload's size. To serve longer prompts on models with a large hidden size, raise it, e.g. to 4294967296 for 131,072 tokens at hidden size 16,384 in bfloat16. |
Prompt-embeds caching¶
Set SG_PROMPT_EMBEDS_CACHE_GIB to a non-zero size and the server keeps the prompt_embeds payloads it is sent, keyed by a content
identifier the client computes with
compute_prompt_embeds_id. A later turn may then send the
identifier in place of the bytes:
A payload the server does not hold is refused with 409 and {"error": {"code": "prompt_embeds_not_cached", "missing_ids": [...]}},
before the engine is involved and before any streaming response has begun. A client has to be able to re-upload on that response; it
is an ordinary part of the protocol, not an error condition. A miss never populates the cache, so a client that treats 409 as fatal
rather than re-uploading will keep missing.
With more than one API server process¶
The cache lives in the API server process. Under --api-server-count N, vLLM runs N of them bound to the same port with SO_REUSEPORT,
and the kernel assigns each TCP connection to one of them, so a client reaches whichever process its connection hashes to. Nothing is
shared between them. Consequences worth sizing for:
- Memory is N times the configured size. The variable bounds one process, so N processes may hold N times that between them. Because each is caching the same payloads, the duplication buys no additional distinct capacity: you spend N×C bytes to get C bytes of caching, and the cache fills up N times sooner than the sizing suggests.
- A payload misses at most N times. Each miss warms exactly one cold process, and a process can only be cold once, so the re-uploads are bounded at N per payload however long the conversation runs.
- Reaching a full hit rate takes longer than that bound. Landing is effectively random per connection, so warming all N processes is
coupon collector: about
N ln(N)turns, which is 3 for N=2, 8 for N=4 and 22 for N=8. Conversations shorter than that still pay occasional re-uploads. - Eviction. The N caches evict independently, so a payload can go cold on one process while staying warm on the others, and its miss count starts again there.
A client that reuses one connection for a conversation avoids all of this, since every turn then reaches the same process. A client that
opens a fresh connection per turn — the usual behaviour behind a load balancer — sees a hit rate of k/N while k of the processes are
warm.
Shared-memory tensor arena¶
With tensor parallelism, every request's prompt embeddings are sent from the engine to each GPU worker. By default vLLM sends each worker
its own copy over a socket, which costs seconds for long prompts: in our measurements, about 2.8 s at TP=2 and 5.4 s at TP=4 for a
256k-token bf16 prompt at hidden size 8192. vLLM's shared-memory tensor arena instead copies the tensor once into /dev/shm, where every
worker maps it and copies it straight to its GPU (about 0.2 s and 0.4 s for the same prompt). The Inference Server enables the arena by
default. It only applies to tensor-parallel servers whose workers share one node; a single-GPU server never uses it.
vLLM's arena has 8 slots of 256 MiB, and a prompt only takes the fast path if its embeddings fit one slot: about 16k tokens in bf16 at
hidden size 8192. For longer prompts, size one slot to your longest prompt, max_model_len × hidden size × 2 bytes in bf16. For example,
for 262,144-token prompts at hidden size 8192:
Two slots are enough for vLLM's default Model Runner V2, which frees a slot about one step after the prompt reaches the workers. Raise
SG_SHM_TENSOR_ARENA_SLOTS if many long prompts arrive at once.
The arena takes slots × slot size of /dev/shm, on top of vLLM's own message queues, which tensor-parallel servers need with or without
the arena:
| Tensor parallelism | vLLM's message queues | Total with a 2 x 4 GiB arena |
|---|---|---|
| TP=2 | 664 MiB | 8.65 GiB |
| TP=4 | 1.1 GiB | 9.12 GiB |
| TP=8 | 2.1 GiB | 10.05 GiB |
With data parallelism, each engine replica on a node builds its own queues and arena, so multiply by --data-parallel-size-local. Leave
headroom for NCCL and PyTorch, which can use /dev/shm too. To give the server enough shared memory:
- With
docker run, pass--ipc=host, or--shm-sizelarger than that total. - With Docker Compose, set
ipc: hostorshm_size:on the service. - On Kubernetes, mount an
emptyDirwithmedium: Memoryat/dev/shm, and set itssizeLimitabove that total.
If /dev/shm is too small for the arena, vLLM refuses to start with an Insufficient space in /dev/shm error. Give the server more
shared memory, lower the two variables above, or pass --no-enable-shm-tensor-arena.
Shared memory counts against the container's memory limit
Pages of /dev/shm the arena touches are charged to the container's memory, like any other tmpfs, and the workers touch all of
them once the first prompt goes through the arena. Include the arena in the container's memory limit.
Privacy
The arena holds prompt embeddings, which are already Stained Glass Transformed, in /dev/shm segments readable only by the server's
user. A slot is not cleared after use: its contents stay in shared memory until a later prompt overwrites them or the server stops.
Prompt embeddings sent as SVD coefficients¶
When the Stained Glass Transform uses a privacy encoder, the Stained Glass Transform Proxy can send each transformed token as its leading
K SVD coefficients instead of a full embedding, either dense or TurboQuant-compressed. The upload shrinks by a factor of
hidden_size / K. The server rebuilds each embedding from its coefficients with the SVD of the privacy encoder's output, the safetensors
file Stained Glass Core's PrivacyEncoderSVD.from_file loads.
Two of the server's plugins decode these requests: stainedglass_privacy_encoder_svd rebuilds the embeddings, and stainedglass_turboquant
first decodes coefficients sent TurboQuant-compressed. vLLM loads every installed plugin unless VLLM_PLUGINS lists the ones to load. If
you set it, list both, and always register_protopia_output_protection, without which the server would return unencrypted output.
To accept these requests:
- Copy the SVD file the proxy is configured with to the server.
- Set
SG_PRIVACY_ENCODER_SVD_PATHto its path on the server.
With Docker, mount the file read-only and point the variable at it, as the highlighted lines of this example do. The other flags are only illustrative. With Docker Compose, add the same volume and environment variable to the service instead.
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v /path/to/privacy_encoder_svd.safetensors:/svd/privacy_encoder_svd.safetensors:ro \
--env "HF_TOKEN=<secret>" \
--env "SG_PRIVACY_ENCODER_SVD_PATH=/svd/privacy_encoder_svd.safetensors" \
-p 8000:8000 \
--ipc=host \
protopia-ai/stainedglass-inference-server:<tag> \
--model <model>
The server checks the file at startup and refuses to start if it is missing or does not hold a valid SVD, then lets the SVD go. The first
request carrying SVD coefficients reads the file again, onto the device that decodes it (the GPU, when there is one), which keeps the only
copy. Leave the file in place while the server runs. If it is gone or holds another SVD by then, those requests are refused, and the
server log names the file, until you restore it or restart the server. Each request carries its own
K, so one server accepts any K from 1 up to the number of singular vectors in the file. TurboQuant compression requires K to be a
power of 2. A request is refused if the file is for embeddings of a size other than the model's hidden size.
TurboQuant compiles its decode once for each K it decodes. Compiling takes seconds, and holds up every other prompt_embeds request
meanwhile. PyTorch also keeps at most eight compiled sizes per process, compressed embeddings included, and decodes any further K
uncompiled, more slowly. Keep K fixed per deployment, and list it in SG_TURBOQUANT_WARMUP_DIMS so that it compiles at startup:
The server must use the same SVD as the proxy: rebuilt with another SVD of the same shape, the coefficients would make wrong embeddings,
and the model's output would degrade without any error. So each request carries the ID of the SVD its coefficients were computed with, a
digest of the SVD's mean and singular vectors, and the server refuses a request whose ID is not its own SVD's, naming both IDs. The server
logs the ID of its SVD at startup (Loaded SVD from ... svd_id=blake3:...), and the proxy logs the ID of its own, so you can compare them
before sending requests.
Privacy
The SVD file holds the mean and leading singular vectors of the privacy encoder's output, computed from Stained Glass Transformed
embeddings of a representative dataset. From K coefficients the server can rebuild only the projection of each transformed
embedding onto the leading K singular vectors, never more than the full transformed embedding it would otherwise receive. The ID of
the SVD only tells SVDs apart, and reveals nothing of the SVD. Errors returned to clients never name the file's path; the server log
does.
Unsupported launch modes¶
Two of vLLM's launch modes never build the OpenAI-compatible FastAPI application that Output Protection encrypts through, so a server started either way would return the model's output in plaintext. Both are refused at startup rather than allowed to serve unprotected:
--grpc, which serves gRPC instead of the OpenAI-compatible HTTP API.VLLM_USE_RUST_FRONTEND=1, which replaces the Python API server with vLLM's Rust frontend.
--headless and --api-server-count 0 are not refused. Those processes serve no HTTP API at all, which is how the engine-only nodes of a multi-node data-parallel deployment run. The nodes of that deployment serving the API are protected normally.