Skip to content

Environment Variables

Configuration can be set using environment variables. Stained Glass Proxy uses Pydantic Settings to manage parsing environment variables.

Most settings are part of the root ProxySettings class, and those settings can be set using the environment variable names listed in the documentation for that attribute, prefixed by SGP_. For example, to set inference_service_host="http://localhost:9600", set the environment variable SGP_INFERENCE_SERVICE_HOST="http://localhost:9600".

Some attributes use a nested settings object, such as sgt_text. To set those nested settings, use the same prefixing strategy, but also include the name of the nested attribute after a double underscore (__). For example, to set sgt_text.path="<my path>", set the environment variable SGP_SGT_TEXT__PATH="<my path>".

.env Files

ProxySettings does not parse a .env file itself; it only reads variables already present in the process environment. Load a .env file into that environment first, then run the proxy: pass uvicorn --env-file <path> (as shown in the README quick start), or first source the file into your shell. If using a Docker Compose deployment, use env_file to load environment variables from a file the same way.

The repository root's .env is git-ignored and local only; copy .env.example to .env and edit it to get started. tests/.env and tests/end2end/.env are different: they are tracked test fixtures with fixed, non-secret values that pytest-dotenv (pytest.ini) and .makefiles/test.mk depend on being present in a fresh checkout, so the test suite and make test work without a manual copy step. They stay tracked rather than being renamed alongside the root .env.

Independent CUDA workers and GPU pools

Independent workers can share a GPU. Set num_workers to the desired number of replicas and optionally set gpu_ids to a JSON list of CUDA-visible GPU ordinals:

```bash
export SGP_SGT_TEXT__DEVICE=cuda
export SGP_SGT_TEXT__GPU_IDS='[0, 1]'
export SGP_SGT_TEXT__NUM_WORKERS=5
```

Workers are assigned round-robin in list order: workers 0, 1, 2, 3, 4 use GPUs 0, 1, 0, 1, 0. Each worker retains a distinct identity in logs and metrics. Omit gpu_ids to use all visible GPUs.

Multiple SGT modalities can use the same GPUs. For example, both text and image workers can use GPU 0:

```bash
export SGP_SGT_TEXT__DEVICE=cuda
export SGP_SGT_TEXT__GPU_IDS='[0]'
export SGP_SGT_TEXT__NUM_WORKERS=10
export SGP_SGT_IMAGE__DEVICE=cuda
export SGP_SGT_IMAGE__GPU_IDS='[0]'
export SGP_SGT_IMAGE__NUM_WORKERS=2
```

A CPU or MPS modality does not require a GPU pool and cannot set gpu_ids.

Visible GPU numbering

IDs refer to the devices visible to the process after CUDA_VISIBLE_DEVICES filtering, not necessarily host GPU IDs. For example, with CUDA_VISIBLE_DEVICES=2,3, the pool [0, 1] selects host GPUs 2 and 3. Each worker verifies at startup, in its own process, that its ordinal is visible and that CUDA selected it, and fails fast otherwise.

Capacity and isolation

Each worker loads its own model; administrators must choose worker counts and GPU pools that fit model weights, activations, and CUDA overhead within the allocator limits below. GPU pools do not enforce isolation between modalities, separate Proxy instances, or unrelated GPU processes.

Device memory budget

Set SGP_SGT_TEXT__GPU_MEMORY_UTILIZATION and SGP_SGT_IMAGE__GPU_MEMORY_UTILIZATION independently to the fraction of each modality's device memory budgeted to its workers. Both default to 0.9. CUDA and CPU accept finite fractions in (0, 1]; MPS accepts (0, 2]. The setting belongs to StainedGlassTransformSettings, not the root settings. Each worker applies its memory limit before distributed initialization and model loading: CUDA uses torch.cuda.memory.set_per_process_memory_fraction, while MPS uses torch.mps.set_per_process_memory_fraction.

For independent workers, the fraction is gpu_memory_utilization / ceil(num_workers / num_pool_gpus), calculated separately for each modality's pool. All workers in a modality receive the same fraction, including those on less populated GPUs:

  • Four workers on one GPU receive 0.9 / 4 = 0.225 each.
  • Eleven workers on two GPUs have a 6/5 distribution; the limit is based on the most crowded GPU in the modality pool so each receive 0.9 / 6 = 0.15. The second GPU's combined allowance is 0.75, not the full 0.9.
  • Text with ten workers on GPU 0 and utilization 0.8 receives 0.08 per worker; image with two workers on GPU 1 and utilization 0.6 receives 0.3 per worker.
export SGP_SGT_TEXT__GPU_MEMORY_UTILIZATION=0.8
export SGP_SGT_IMAGE__GPU_MEMORY_UTILIZATION=0.6

Each tensor-parallel rank receives the full configured fraction on its distinct GPU.

To disable worker memory limits entirely, set SGP_ENABLE_WORKER_MEMORY_LIMITS=false (default true). Workers then leave the backend allocator limits unchanged, and gpu_memory_utilization is ignored.

When multiple SGT modalities have overlapping GPU pools (including an omitted gpu_ids, which means all visible GPUs), startup logs a warning and leaves allocator limits unchanged for those modalities only. The configured enable_worker_memory_limits setting is not modified, and non-overlapping modalities retain their limits. Per-modality fractions cannot guarantee a safe combined budget on a shared GPU. Assign disjoint gpu_ids pools to apply limits to both modalities.

CPU and MPS modalities are each treated as a single-device pool: a modality with four workers receives 0.9 / 4 = 0.225 per worker. CPU memory limits are not supported, so applying a fraction emits a warning without enforcing a limit. If two modalities share CPU or share MPS, no fraction is set for either modality; startup logs a warning that limiting memory for two modalities on the same device is not yet supported. Separate CPU and MPS modalities do not count as sharing.

These are PyTorch backend allocator limits, not VRAM reservations or hard limits on every GPU allocation. CUDA contexts, libraries, and other processes can consume additional memory, and allocations can fail before reaching the limit when free memory is insufficient. CUDA uses total device capacity, while MPS uses its recommended maximum working-set size, not current free memory. Equal fractions imply equal byte allowances only on GPUs with equal capacity. Exceeding a worker's allowance raises a CUDA out-of-memory error even if its GPU still has unused memory; there is no automatic borrowing or resizing. A worker is also free to use less than its calculated allowance, if it never needs to allocate that much memory.

MPS workers emit a warning when setting a limit: Metal's recommended working-set size can be substantially less than total unified memory, depending on the system. On MPS, 1.0 means the recommended maximum, not the physical memory limit. Values up to 2.0 follow PyTorch's MPS allocator range; zero remains rejected to avoid unlimited allocations. For example, an MPS modality configured with utilization 1.5 and two workers gives each worker 0.75 of the recommended working set. For reference, PyTorch hard-codes a default maximum allocation of 1.7 * Apple's recommendedMaxWorkingSetSize (i.e. equivalent to setting this value to 1.7 and having a single worker). Additionally, empirical evidence seems to suggest that recommendedMaxWorkingSetSize is usually around 74-85% of the total unified memory, but your particular system may be higher or lower than that. Exceeding 1.0 increases chances of system instability because CPU resources will have to compete for memory with the GPU. Higher limits do not guarantee that the system has enough physical memory.

Tensor parallelism keeps its existing placement and single-replica restriction. Do not set gpu_ids with tensor parallelism. Configurations combining tensor parallelism with another CUDA modality are rejected; use separate Proxy instances instead.

ProxySettings

Settings for Stained Glass Proxy.

Methods:

Name Description
per_modality_gpu_memory_fractions

Calculate uniform per-worker allocator fractions without changing configured settings.

settings_customise_sources

Define the sources and their order for loading the settings values.

Attributes:

Name Type Description
inference_service_host str

The hostname of the upstream service.

sgt_text StainedGlassTransformSettings | None

Settings for the Stained Glass Transform for text inputs.

sgt_image StainedGlassTransformSettings | None

Settings for the Stained Glass Transform for image inputs.

modality_allow_pass_through ModalityAllowPassthroughSettings

Settings for allowing passthrough of message parts to the upstream inference service.

readiness_thresholds ReadinessThresholdSettings

Thresholds controlling when /readyz reports the proxy as NOT READY (HTTP 503). See ReadinessThresholdSettings.

shared_memory_min_available_bytes int

Warn when constructing settings on Linux if available /dev/shm space is below this byte threshold.

default_sampling_params DefaultSamplingParams

Default sampling parameters for generation when not specified in the request.

seed int | None

The seed for Stained Glass Transform and inference.

enable_worker_memory_limits bool

Whether SGT workers apply per-process allocator memory limits derived from each modality's gpu_memory_utilization.

allow_request_seed_for_sgt bool

Whether requests may carry a seed that is applied to Stained Glass Transform.

upstream_keep_alive_timeout float

Timeout for idle connections with the upstream inference server.

upstream_connector_limit int

Maximum simultaneous upstream HTTP connections for aiohttp upstream manager.

session_timeout float

Timeout for connections with the upstream inference server.

api_username str | None

The username for upstream inference server authentication.

api_password str | None

The password for upstream inference server authentication.

use_aiohttp_for_upstream bool

Prefer using aiohttp over httpx to manage http connections with the upstream inference server. This may have improved throughput

upstream_follow_redirects bool

Follow 3xx redirects from the upstream inference server, at the cost of the streamed request body.

stream_upstream_request_body bool

Stream large request bodies (base64 prompt_embeds payloads) to the upstream inference server in chunks

bypass_upstream_version_checks bool

Skip startup upstream version probes and version-gated capability checks.

upstream_use_chat_completions_for_prompt_embeds bool

Route transformed prompt embeddings through vLLM's /v1/chat/completions endpoint as prompt_embeds content parts, instead of

prompt_embeds_cache_enabled bool

Reference prompt_embeds content parts by content identifier instead of uploading them, re-uploading only when there is an upstream cache miss.

cache_salt str | None

Default cache_salt for upstream requests that do not carry their own.

max_embed_decode_bytes int

Maximum bytes a client-supplied prompt_embeds payload may allocate once densified (num_elements * element_size).

sagemaker_endpoint_name str | None

Name of AWS SageMaker AI endpoint to use as the upstream LLM Inference server. Should be None unless using a SageMaker endpoint.

sagemaker_region_name str | None

AWS region name for the SageMaker endpoint. Should be None unless using a SageMaker endpoint.

max_input_tokens int | None

The maximum number of input tokens to allow in a /v1/chat/completions or /v1/stainedglass request.

transform_system_prompt bool | None

Whether to apply Stained Glass Transform to system-role messages.

output_decryption bool

Whether to decrypt the output from the upstream service.

ephemeral_key_refresh_time_seconds float

The time in seconds to refresh the ephemeral key, when output decryption is enabled.

client_public_key_header_name str

The name of the HTTP header to use for passing the client public key to the upstream inference server.

server_public_key_header_name str

The name of the HTTP header to use for parsing the server public key from the downstream client.

warmup_strategy WarmupStrategy | None

How far through the inference pipeline to exercise during the startup warmup, so the first real request does not

tool_parser str | None

The tool parser to use (see vllm serve -h | grep tool-call-parser). If None, tool calls are not parsed Proxy-side.

reasoning_parser str | None

The reasoning parser to use (see vllm serve -h | grep reasoning-parser). If None, reasoning content is not split out Proxy-side.

custom_chat_template_file str | None

Path to a file containing a custom Jinja chat template. If provided, this takes precedence over custom_chat_template.

default_chat_template_kwargs Annotated[dict[str, Any] | None, NoDecode]

This is especially useful to enable or disable thinking for chat templates which support the feature.

tokenizer_for_parsers str | None

Tokenizer to load from Hugging Face Hub (or local directory) for use with tool call/reasoning parsers.

profile bool

Enable profiling for the proxy. Enabling this will create a folder for profiling data if it does not already exist.

profile_data_folder Path

The folder to store profiling data. This is only used if profiling is enabled.

logging_config_file str

The logging configuration file.

license_manager_configuration_file Path

The path to the license manager configuration file.

allowed_headers Annotated[list[str], NoDecode]

List of allowed headers that will be forwarded on to the downstream service.

cors_allow_origins Annotated[list[str], NoDecode]

List of allowed origins for CORS. This is expected to be a list of URLs.

cors_allow_headers Annotated[list[str], NoDecode]

List of allowed headers for CORS. This is expected to be a list of http headers.

request_headers_to_add Annotated[dict[str, str] | None, NoDecode]

A dictionary of headers to add to each request to the upstream inference service.

request_headers_to_log Annotated[list[str] | None, NoDecode]

List of request headers to include in log messages for each request.

enable_opentelemetry bool

Whether to enable OpenTelemetry instrumentation for the proxy.

sgt_names dict[Modality, str | None]

Return a mapping of modality enums (e.g. Modality.TEXT, Modality.IMAGE) to their configured SGT name, or None if unset.

inference_service_host instance-attribute

inference_service_host: str

The hostname of the upstream service.

This should include the protocol (http or https) and the port if not the default.

Examples:

  • http://localhost:8000
  • https://example.com:443
  • http://vllm:8080

sgt_text class-attribute instance-attribute

sgt_text: StainedGlassTransformSettings | None = None

Settings for the Stained Glass Transform for text inputs.

sgt_image class-attribute instance-attribute

sgt_image: StainedGlassTransformSettings | None = None

Settings for the Stained Glass Transform for image inputs.

modality_allow_pass_through class-attribute instance-attribute

modality_allow_pass_through: ModalityAllowPassthroughSettings = pydantic.Field(
    default_factory=ModalityAllowPassthroughSettings
)

Settings for allowing passthrough of message parts to the upstream inference service.

readiness_thresholds class-attribute instance-attribute

readiness_thresholds: ReadinessThresholdSettings = (
    pydantic.Field(
        default_factory=ReadinessThresholdSettings
    )
)

Thresholds controlling when /readyz reports the proxy as NOT READY (HTTP 503). See ReadinessThresholdSettings.

shared_memory_min_available_bytes class-attribute instance-attribute

shared_memory_min_available_bytes: int = pydantic.Field(
    default=2 * 1024**3, ge=0
)

Warn when constructing settings on Linux if available /dev/shm space is below this byte threshold.

Defaults to 2 GiB. Set to zero to disable the check. Provision at least 2 GiB per concurrent request and raise this threshold for expected concurrency. This check does not reserve memory or enforce admission limits.

default_sampling_params class-attribute instance-attribute

default_sampling_params: DefaultSamplingParams = (
    DefaultSamplingParams()
)

Default sampling parameters for generation when not specified in the request.

seed class-attribute instance-attribute

seed: int | None = None

The seed for Stained Glass Transform and inference.

enable_worker_memory_limits class-attribute instance-attribute

enable_worker_memory_limits: bool = True

Whether SGT workers apply per-process allocator memory limits derived from each modality's gpu_memory_utilization.

Set to False (SGP_ENABLE_WORKER_MEMORY_LIMITS=false) to leave backend allocator limits unchanged for all workers. This is forced to False, with a warning, when multiple CUDA modalities have overlapping GPU pools.

allow_request_seed_for_sgt class-attribute instance-attribute

allow_request_seed_for_sgt: bool = True

Whether requests may carry a seed that is applied to Stained Glass Transform.

A seeded transform is deterministic, which defeats the stochastic noise SGT relies on for protection, so every seeded request is logged as a warning.

Disable this to refuse seeded requests, for a deployment that must not serve reproducible transforms at all. Requests that include seed are then rejected with HTTP 422.

upstream_keep_alive_timeout class-attribute instance-attribute

upstream_keep_alive_timeout: float = 5

Timeout for idle connections with the upstream inference server.

upstream_connector_limit class-attribute instance-attribute

upstream_connector_limit: int = pydantic.Field(
    default=100, ge=0
)

Maximum simultaneous upstream HTTP connections for aiohttp upstream manager.

This maps to aiohttp TCPConnector limit. The default (100) matches aiohttp's default. This value must be greater than or equal to 0. A value of 0 disables the connection limit (i.e., no limit is enforced).

Insufficient connector limits cause two problems
  1. Queueing latency: Requests wait in the proxy's internal queue before acquiring a connection.
  2. Broken pipe risk: Queued requests may reuse pooled connections that are stale or right at the keep-alive timeout boundary. If the upstream closes the connection while it sits in the queue, the request fails with connection errors.
Safety Constraints
  • Do not exceed OS file descriptor limit (check ulimit -n).
  • Do not significantly exceed upstream's own concurrency budget.
  • Memory per connection: Each active connection consumes memory. Monitor and adjust based on available resources and traffic patterns.

session_timeout class-attribute instance-attribute

session_timeout: float = 600

Timeout for connections with the upstream inference server.

api_username class-attribute instance-attribute

api_username: str | None = pydantic.Field(
    default=None, repr=False
)

The username for upstream inference server authentication.

api_password class-attribute instance-attribute

api_password: str | None = pydantic.Field(
    default=None, repr=False
)

The password for upstream inference server authentication.

use_aiohttp_for_upstream class-attribute instance-attribute

use_aiohttp_for_upstream: bool = True

Prefer using aiohttp over httpx to manage http connections with the upstream inference server. This may have improved throughput when handling many concurrent requests.

upstream_follow_redirects class-attribute instance-attribute

upstream_follow_redirects: bool = False

Follow 3xx redirects from the upstream inference server, at the cost of the streamed request body.

Applies to the aiohttp client only (use_aiohttp_for_upstream, the default).

By default the aiohttp client streams each request body to the socket a chunk at a time, which keeps peak SGT Proxy memory low under concurrent long-context requests. A streamed body is consumed as it is sent, so it cannot be replayed on the second request a redirect requires, redirect responses are therefore refused with a 502.

Enable this for an upstream that answers with redirects. Request bodies are then serialized in full before being sent, so the HTTP client can replay them, and the memory savings of the streamed body are lost.

stream_upstream_request_body class-attribute instance-attribute

stream_upstream_request_body: bool = False

Stream large request bodies (base64 prompt_embeds payloads) to the upstream inference server in chunks instead of fully materializing the JSON body before sending.

This substantially reduces peak SGT Proxy memory under concurrent long-context requests. This requires the HTTP client to send the body with Transfer-Encoding: chunked, which not every upstream deployment handles correctly. Notably, Modal-hosted upstreams break on chunked requests. Leave this False (the default) for Modal-hosted or otherwise chunked-transfer-encoding-incompatible upstreams. Enable it only for upstreams verified to handle chunked request bodies correctly.

Note: This now applies to the httpx client only, and is scheduled for removal once httpx adopts the same approach. The aiohttp client (use_aiohttp_for_upstream) always streams its request bodies and always frames them with Content-Length, so it needs no opt-in and is unaffected by upstreams that reject chunked requests.

bypass_upstream_version_checks class-attribute instance-attribute

bypass_upstream_version_checks: bool = False

Skip startup upstream version probes and version-gated capability checks.

Set SGP_BYPASS_UPSTREAM_VERSION_CHECKS=true only when upstream compatibility has been verified independently. This does not change feature routing or make unsupported features compatible; incompatible requests may fail at runtime.

upstream_use_chat_completions_for_prompt_embeds class-attribute instance-attribute

upstream_use_chat_completions_for_prompt_embeds: bool = True

Route transformed prompt embeddings through vLLM's /v1/chat/completions endpoint as prompt_embeds content parts, instead of converting to a legacy /v1/completions request.

Requires a vLLM upstream that supports prompt_embeds in chat messages (>= v0.21.0). Leave False for backwards compatibility with older vLLM versions.

Setting this to False is now deprecated, and lacks support for many features such as reasoning and multi-modal.

prompt_embeds_cache_enabled class-attribute instance-attribute

prompt_embeds_cache_enabled: bool = False

Reference prompt_embeds content parts by content identifier instead of uploading them, re-uploading only when there is an upstream cache miss.

Each transformed part is first sent as a content-based identifier alone. An upstream that does not hold it answers 409 prompt_embeds_not_cached with every missing identifier, and the request is re-sent with those parts uploaded.

Requires upstream_use_chat_completions_for_prompt_embeds, a text SGT that transforms in chunks (sgt_text.chunk_size), and an output-protected vLLM upstream (Stained Glass Output Protection >= 2.38.0) started with SG_PROMPT_EMBEDS_CACHE_GIB set to a non-zero size.

Warning

A cache hit needs byte-identical embeddings. Stained Glass Transform draws fresh noise on every request. Enable this only where the embeddings are deterministic. For now the only way to get deterministic embeddings is SGP__DEBUG_CLEAN_EMBEDS=true, which bypasses Stained Glass Transform entirely and is meant for testing only.

cache_salt class-attribute instance-attribute

cache_salt: str | None = pydantic.Field(
    default=None, min_length=1, max_length=1024, repr=False
)

Default cache_salt for upstream requests that do not carry their own.

vLLM salts its prefix cache with it, and an output-protected upstream also files cached prompt_embeds payloads under a scope named by it. Requests carrying different salts share neither, and requests without a salt all share one scope.

Any string of 1 to 1024 characters, used as is: it is never decoded, so no particular encoding is required. It should be random, unpredictable and kept secret.

max_embed_decode_bytes class-attribute instance-attribute

max_embed_decode_bytes: int = 2147483648

Maximum bytes a client-supplied prompt_embeds payload may allocate once densified (num_elements * element_size).

A sparse tensor carries its own declared shape, so a payload of only a few KB can declare a shape that would require hundreds of GiB once densified via torch.Tensor.to_dense(). This limit is checked before densification happens, so the memory is never allocated for oversized payloads; the request is rejected with HTTP 422 instead.

Defaults to 2 GiB, matching vLLM's VLLM_MAX_EMBED_DECODE_BYTES default. In terms of legitimate traffic, 2 GiB admits 2**31 / (hidden_size * 4) float32 tokens per request: roughly 131k tokens at hidden_size=4096 (8B-class models) but only 65k at hidden_size=8192 (70B-class). Raise it when serving a large model at a context length beyond that, and size it against available host RAM, since the decode is a CPU allocation in the proxy process.

Set to 0 to disable this limit, which is not recommended for deployments exposed to untrusted users.

sagemaker_endpoint_name class-attribute instance-attribute

sagemaker_endpoint_name: str | None = None

Name of AWS SageMaker AI endpoint to use as the upstream LLM Inference server. Should be None unless using a SageMaker endpoint.

When set, inference_service_host must be set to the empty string, otherwise, the proxy will throw an error while loading.

sagemaker_region_name class-attribute instance-attribute

sagemaker_region_name: str | None = None

AWS region name for the SageMaker endpoint. Should be None unless using a SageMaker endpoint.

When set, sagemaker_endpoint_name must also be set, otherwise, the proxy will throw an error while loading.

max_input_tokens class-attribute instance-attribute

max_input_tokens: int | None = None

The maximum number of input tokens to allow in a /v1/chat/completions or /v1/stainedglass request.

Requests with token count greater than this value will be rejected with a 413 error code.

transform_system_prompt class-attribute instance-attribute

transform_system_prompt: bool | None = None

Whether to apply Stained Glass Transform to system-role messages.

If set to True or False, this setting overrides any per-request value provided in chat completion request extra body. If None, per-request value is used when provided.

output_decryption class-attribute instance-attribute

output_decryption: bool = False

Whether to decrypt the output from the upstream service.

Warning

This should only be enabled when the upstream inference service uses output encryption.

ephemeral_key_refresh_time_seconds class-attribute instance-attribute

ephemeral_key_refresh_time_seconds: float = 15 * 60

The time in seconds to refresh the ephemeral key, when output decryption is enabled.

client_public_key_header_name class-attribute instance-attribute

client_public_key_header_name: str = 'x-client-public-key'

The name of the HTTP header to use for passing the client public key to the upstream inference server.

server_public_key_header_name class-attribute instance-attribute

server_public_key_header_name: str = 'x-server-public-key'

The name of the HTTP header to use for parsing the server public key from the downstream client.

warmup_strategy class-attribute instance-attribute

warmup_strategy: WarmupStrategy | None = None

How far through the inference pipeline to exercise during the startup warmup, so the first real request does not pay the initialization (first-call) cost.

  • none: send no warmup request.
  • sgt: warm up only the SGT via chat_to_chat (does not forward to the upstream). Triggers torch.compile only when compile_noise_layer_forward is enabled.
  • end_to_end: send a full chat completion through the Proxy/SGT -> Upstream path (also triggers torch.compile when compile_noise_layer_forward is enabled).

If None (default), a strategy is chosen to preserve the existing behavior: sgt when the noise layer forward pass is compiled (sgt_text.compile_noise_layer_forward), otherwise none. Set this explicitly to warm up independently of the compile setting. For example, end_to_end to absorb the upstream first-call cost, or sgt for models that do not support torch.compile but still incur a first-call cost.

tool_parser class-attribute instance-attribute

tool_parser: str | None = None

The tool parser to use (see vllm serve -h | grep tool-call-parser). If None, tool calls are not parsed Proxy-side.

Proxy-side parsing runs on the plaintext model output: - Always on the chat->completions path (the upstream /v1/completions endpoint never parses). - On the chat->chat path only when Proxy actually decrypted the output (output-decryption is True): The output-protected upstream returns the encrypted output in content without parsing, so Proxy parses the decrypted plaintext. A non-output-protected upstream already parsed server-side, so in that case Proxy does not parse.

reasoning_parser class-attribute instance-attribute

reasoning_parser: str | None = None

The reasoning parser to use (see vllm serve -h | grep reasoning-parser). If None, reasoning content is not split out Proxy-side.

Proxy-side parsing runs on the plaintext model output: - Always on the chat->completions path, since the upstream /v1/completions endpoint never parses. - On the chat->chat path only when Proxy actually decrypted the output (output-decryption is True): The output-protected upstream returns the encrypted output in content without parsing, so Proxy parses the decrypted plaintext. A non-output-protected upstream already parsed server-side, so in that case Proxy does not parse.

custom_chat_template_file class-attribute instance-attribute

custom_chat_template_file: str | None = None

Path to a file containing a custom Jinja chat template. If provided, this takes precedence over custom_chat_template.

default_chat_template_kwargs class-attribute instance-attribute

default_chat_template_kwargs: Annotated[
    dict[str, Any] | None, NoDecode
] = None

This is especially useful to enable or disable thinking for chat templates which support the feature.

Examples:

>>> SGP_DEFAULT_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "conversation_id": "abc123"}'
>>> SGP_DEFAULT_CHAT_TEMPLATE_KWARGS='{"enable_thinking": false, "conversation_id": "def456"}'

tokenizer_for_parsers class-attribute instance-attribute

tokenizer_for_parsers: str | None = None

Tokenizer to load from Hugging Face Hub (or local directory) for use with tool call/reasoning parsers.

If None (the default), it extracts the tokenizer from the Stained Glass Transform for Text (sgt_text) and uses that tokenizer for the parsers. If a tokenizer is specified, it will be loaded using transformers.AutoTokenizer.from_pretrained.

It is an error to specify both this setting and a Stained Glass Transform for Text (sgt_text).

This setting can be useful when transforming only, say, image inputs with an SGT, but still submitting a conversation with plaintext text messages and tool calls to the upstream inference service. This setting will let proxy load a tokenizer for the tool call/reasoning parsers without needing to load a Stained Glass Transform for Text.

profile class-attribute instance-attribute

profile: bool = False

Enable profiling for the proxy. Enabling this will create a folder for profiling data if it does not already exist.

profile_data_folder class-attribute instance-attribute

profile_data_folder: Path = pathlib.Path('profile')

The folder to store profiling data. This is only used if profiling is enabled.

logging_config_file class-attribute instance-attribute

logging_config_file: str = 'logging.yaml'

The logging configuration file.

license_manager_configuration_file class-attribute instance-attribute

license_manager_configuration_file: Path = pathlib.Path(
    "license_manager_configuration.json"
)

The path to the license manager configuration file.

allowed_headers class-attribute instance-attribute

allowed_headers: Annotated[list[str], NoDecode] = []

List of allowed headers that will be forwarded on to the downstream service.

This is expected to be a comma separated list. ie 'Modal-Key, Modal-Secret'.

cors_allow_origins class-attribute instance-attribute

cors_allow_origins: Annotated[list[str], NoDecode] = ['*']

List of allowed origins for CORS. This is expected to be a list of URLs. The default is to allow all origins.

cors_allow_headers class-attribute instance-attribute

cors_allow_headers: Annotated[list[str], NoDecode] = ['*']

List of allowed headers for CORS. This is expected to be a list of http headers. The default is to allow all headers.

request_headers_to_add class-attribute instance-attribute

request_headers_to_add: Annotated[
    dict[str, str] | None, NoDecode
] = pydantic.Field(default=None, repr=False)

A dictionary of headers to add to each request to the upstream inference service.

request_headers_to_log class-attribute instance-attribute

request_headers_to_log: Annotated[
    list[str] | None, NoDecode
] = None

List of request headers to include in log messages for each request.

This is expected to be a comma separated list. ie 'X-Request-ID, User-Agent'. Only headers present in the request will be logged.

enable_opentelemetry class-attribute instance-attribute

enable_opentelemetry: bool = False

Whether to enable OpenTelemetry instrumentation for the proxy.

sgt_names property

sgt_names: dict[Modality, str | None]

Return a mapping of modality enums (e.g. Modality.TEXT, Modality.IMAGE) to their configured SGT name, or None if unset.

per_modality_gpu_memory_fractions

per_modality_gpu_memory_fractions() -> dict[
    Modality, float | None
]

Calculate uniform per-worker allocator fractions without changing configured settings.

CUDA fractions account for round-robin replica placement; tensor-parallel ranks each receive the full configured fraction. CPU and MPS are each treated as a single device. Modalities sharing a device receive no limit and emit a warning.

Returns:

Type Description
dict[Modality, float | None]

Every modality's fraction, or None if unconfigured, limits are disabled, or its device is shared.

settings_customise_sources classmethod

settings_customise_sources(
    settings_cls: type[BaseSettings],
    init_settings: PydanticBaseSettingsSource,
    env_settings: PydanticBaseSettingsSource,
    dotenv_settings: PydanticBaseSettingsSource,
    file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]

Define the sources and their order for loading the settings values.

Parameters:

Name Type Description Default
settings_cls type[BaseSettings]

The Settings class.

required
init_settings PydanticBaseSettingsSource

The InitSettingsSource instance.

required
env_settings PydanticBaseSettingsSource

The EnvSettingsSource instance.

required
dotenv_settings PydanticBaseSettingsSource

The DotEnvSettingsSource instance.

required
file_secret_settings PydanticBaseSettingsSource

The SecretsSettingsSource instance.

required

Returns:

Type Description
tuple[PydanticBaseSettingsSource, ...]

A tuple containing the sources and their order for loading the settings values.

StainedGlassTransformSettings

Settings related to the Stained Glass Transform itself.

Attributes:

Name Type Description
path str

Path to the Stained Glass Transform file.

name str | None

Optional name to use for the SGT in the /models endpoint and as the model id for /v1/chat/completions and /v1/completions requests.

device Literal['cpu', 'cuda', 'mps']

Device that Stained Glass Transform will run on.

gpu_memory_utilization float

Fraction of this modality's device memory budgeted to its workers (default 0.9).

gpu_ids list[Annotated[int, Field(strict=True, ge=0)]] | None

CUDA-visible GPU IDs for independent workers, in round-robin order.

num_workers int

The number of SGT workers.

use_cache bool | None

The value of use_cache to pass to the modeling blocks underlying the SGT mean and std estimator modules.

chunk_size Annotated[PositiveInt | None, BeforeValidator(_none_from_env)]

The number of tokens per chunk of the text SGT's chunked forward pass, or None (e.g. SGP_SGT_TEXT__CHUNK_SIZE=none) to

tensor_parallel_size int | None

Tensor parallel size (must be a power of 2, min 2, max num_gpus). If None, disables tensor parallelism.

grace_period_seconds float

The grace period in seconds to wait for workers to shutdown.

worker_ready_timeout_seconds float | None

The timeout in seconds to wait for workers to signal ready during startup. If None, waits indefinitely.

worker_stats_interval_seconds float

How often, in seconds, each SGT worker process reports its CUDA memory usage back to the manager for the /load endpoint.

metrics_task_history_size int

Number of recent completed tasks to retain (per worker, and for the modality-wide latency window) for rolling error-rate and

metrics_error_rate_window_minutes float

Size, in minutes, of the trailing time window each worker's GPU/CUDA error rate is computed over for

torch_dtype str | None

The data type for the Stained Glass Transform. If None, uses the dtype that the SGT's weights are saved in.

noise_layer_attention SupportedAttentionImplementationsType

The attention mechanism for the noise layer. If None, uses the attention mechanism saved in the SGT file.

third_party_model_path str | PathLike[str] | None

The path or huggingface reference to a third-party model to load. This is useful when loading SGTs whose internal structure depends

trust_remote_code bool

Whether to trust remote code when loading from HuggingFace Hub.

compile_noise_layer_forward bool | None

Whether to compile the noise layer's forward function.

override_transform_all_tokens bool | None

Override the transform_all_tokens setting from the SGT file.

embedding_compression EmbeddingCompressionStrategy

Compression algorithm for SGT embeddings sent to the upstream vLLM inference server.

embedding_compression_bits Annotated[Literal[1, 2, 3, 4, 8], BeforeValidator(int)]

Quantization bit-width used when embedding_compression is turboquant.

embedding_compression_turboquant_chunk_size int

Chunk size used when embedding_compression is turboquant.

prompt_embeds_serialization_threads PositiveInt

Number of content parts each text SGT worker serializes concurrently for chat requests forwarded as prompt_embeds.

path instance-attribute

path: str

Path to the Stained Glass Transform file.

This can be a path to a .sgt zipfile or a model name on the Hugging Face Hub (such as Protopia/SGT-for-llama-3.1-8b-instruct-rare-rain-bfloat16). Passing a local directory is not supported.

name class-attribute instance-attribute

name: str | None = None

Optional name to use for the SGT in the /models endpoint and as the model id for /v1/chat/completions and /v1/completions requests.

When this is set, the model can be referred to either as <base model name> or <base model name>/<sgt name>, and both options will be visible from the /models endpoint.

device class-attribute instance-attribute

device: Literal['cpu', 'cuda', 'mps'] = 'cpu'

Device that Stained Glass Transform will run on.

gpu_memory_utilization class-attribute instance-attribute

gpu_memory_utilization: float = pydantic.Field(
    default=0.9, gt=0, le=2, allow_inf_nan=False
)

Fraction of this modality's device memory budgeted to its workers (default 0.9).

Accepts (0, 1] for CUDA/CPU and (0, 2] for MPS. Independent workers divide this by the maximum workers per device in the modality's pool; each TP rank gets the full fraction. CPU/MPS are single-device pools. CPU limits only warn; sharing CPU or MPS across modalities skips limits with a warning. CUDA fractions use total VRAM, while MPS fractions use Metal's recommended maximum working-set size, not physical unified memory. These are allocator limits, not reservations.

gpu_ids class-attribute instance-attribute

gpu_ids: (
    list[Annotated[int, Field(strict=True, ge=0)]] | None
) = None

CUDA-visible GPU IDs for independent workers, in round-robin order.

Omit to use all visible GPUs. Pools may overlap across modalities. Not supported with CPU, MPS, or tensor parallelism. IDs respect CUDA_VISIBLE_DEVICES.

num_workers class-attribute instance-attribute

num_workers: int = 1

The number of SGT workers.

Increasing this may improve throughput when handling many concurrent requests, but will also increase memory usage.

use_cache class-attribute instance-attribute

use_cache: bool | None = None

The value of use_cache to pass to the modeling blocks underlying the SGT mean and std estimator modules.

chunk_size class-attribute instance-attribute

chunk_size: Annotated[
    PositiveInt | None, BeforeValidator(_none_from_env)
] = 4096

The number of tokens per chunk of the text SGT's chunked forward pass, or None (e.g. SGP_SGT_TEXT__CHUNK_SIZE=none) to transform each request in a single pass. The image SGT ignores it.

A chunked pass transforms a request sequentially, chunk_size tokens at a time, carrying the estimators' state from one chunk to the next. Peak GPU memory then grows with one chunk's activations instead of the whole request's, and a seeded transform becomes prefix-stable: the earlier messages of a growing conversation are transformed identically on every turn, so the prompt-embeds cache can serve them. An unseeded transform still draws fresh noise for every request.

Seeded outputs depend on chunk_size, so it is effectively part of the seed. Stained Glass Core rejects chunked passes for SGTs it cannot chunk.

tensor_parallel_size class-attribute instance-attribute

tensor_parallel_size: int | None = None

Tensor parallel size (must be a power of 2, min 2, max num_gpus). If None, disables tensor parallelism.

grace_period_seconds class-attribute instance-attribute

grace_period_seconds: float = 5

The grace period in seconds to wait for workers to shutdown.

worker_ready_timeout_seconds class-attribute instance-attribute

worker_ready_timeout_seconds: float | None = None

The timeout in seconds to wait for workers to signal ready during startup. If None, waits indefinitely.

worker_stats_interval_seconds class-attribute instance-attribute

worker_stats_interval_seconds: float = pydantic.Field(
    default=15.0, gt=0
)

How often, in seconds, each SGT worker process reports its CUDA memory usage back to the manager for the /load endpoint.

metrics_task_history_size class-attribute instance-attribute

metrics_task_history_size: int = pydantic.Field(
    default=100, gt=0
)

Number of recent completed tasks to retain (per worker, and for the modality-wide latency window) for rolling error-rate and latency estimates reported by the /load endpoint. A count-based window, not a time-based one: a worker that goes idle keeps showing whatever rate its last metrics_task_history_size tasks produced.

metrics_error_rate_window_minutes class-attribute instance-attribute

metrics_error_rate_window_minutes: float = pydantic.Field(
    default=5.0, gt=0, allow_inf_nan=False
)

Size, in minutes, of the trailing time window each worker's GPU/CUDA error rate is computed over for /load's WorkerMetrics.cuda_error_rate_trailing_window and /readyz's gpu_error_rate threshold check (ReadinessThresholdSettings.gpu_readiness.max_error_rate). Unlike metrics_task_history_size (a count-based window backing WorkerMetrics.cuda_error_rate), this is time-based: an idle worker's rate decays toward None as outcomes age out of the window, rather than indefinitely reflecting a stale batch of tasks.

torch_dtype class-attribute instance-attribute

torch_dtype: str | None = None

The data type for the Stained Glass Transform. If None, uses the dtype that the SGT's weights are saved in.

For most use cases, we recommend this be "torch.bfloat16". If you are seeing unexpectedly large memory consumption, try explicitly setting this option.

noise_layer_attention class-attribute instance-attribute

noise_layer_attention: SupportedAttentionImplementationsType = None

The attention mechanism for the noise layer. If None, uses the attention mechanism saved in the SGT file.

For most use cases, we recommend this to be "flash_attention_2". If you are seeing unexpectedly large memory consumption, try explicitly setting this option.

Warning

Not all attention mechanisms may be available for all dtypes and devices.

third_party_model_path class-attribute instance-attribute

third_party_model_path: str | PathLike[str] | None = None

The path or huggingface reference to a third-party model to load. This is useful when loading SGTs whose internal structure depends on transformers which are not importable directly through transformers, but are present on the Hugging Face Hub. Typically this should be None or False.

trust_remote_code class-attribute instance-attribute

trust_remote_code: bool = False

Whether to trust remote code when loading from HuggingFace Hub.

Warning

Enabling this allows execution of arbitrary code from the model repository on Hugging Face Hub. Only enable this for models from trusted sources.

compile_noise_layer_forward class-attribute instance-attribute

compile_noise_layer_forward: bool | None = None

Whether to compile the noise layer's forward function.

Compiling the forward function can provide around a 2x speedup when using CUDA. On MPS, this feature is still experimental, and enabling it is not recommended until it becomes more stable.

override_transform_all_tokens class-attribute instance-attribute

override_transform_all_tokens: bool | None = None

Override the transform_all_tokens setting from the SGT file.

"Transform all tokens" refers to whether or not the SGT is applied to all tokens in the input, including the chat template itself, or if it is just applied to user-provided inputs (such as message contents).

When upstream_use_chat_completions_for_prompt_embeds and transform_all_tokens are both True, although the SGT is in fact applied to all tokens, only the user-provided inputs are extracted and sent to the upstream as prompt_embeds content parts. When the chat template is reassembled in the upstream provider, the chat template will effectively be untransformed (but the messages will be still sent as prompt_embeds). Consequently, the transform_all_tokens setting does not really have much of an effect.

This setting is helpful when using the /v1/chat/completions endpoint with upstream_use_chat_completions_for_prompt_embeds=False. In that case, the SGT is applied to the entire prompt, including the chat template, and the entire transformed prompt_embeds are sent to the upstream's /v1/completions endpoint. In that case, the upstream does not do any chat template manipulation, so its inputs are truly transformed before hitting the model.

This setting is also particularly useful when using the /v1/completions endpoint. In that case, the SGT has no way of calculating a noise mask to determine which tokens are user-provided and which are from the chat template, so the SGT will simply transform everything. If the SGT file settings dictate that transform_all_tokens=False, SGT Proxy will throw an error on the request to avoid transforming all of the tokens in an undefined way (since the SGT was trained without transforming chat template tokens). This override setting allows you to still use that SGT with the /v1/completions endpoint, albeit with the caveat that all tokens will indeed be transformed, which may result in worse generation quality if the SGT was not trained with transforming all tokens.

If None, the transform_all_tokens setting from the SGT file will be used. If True or False, this setting will take precedence over the SGT file's setting.

embedding_compression class-attribute instance-attribute

embedding_compression: EmbeddingCompressionStrategy = (
    EmbeddingCompressionStrategy.NONE
)

Compression algorithm for SGT embeddings sent to the upstream vLLM inference server.

When set to anything other than none, the proxy encodes embeddings before sending them over the wire. The vLLM server must have the stainedglass_output_protection TurboQuant plugin active (stainedglass_turboquant registered under vllm.general_plugins) to decode them.

embedding_compression_bits class-attribute instance-attribute

embedding_compression_bits: Annotated[
    Literal[1, 2, 3, 4, 8], BeforeValidator(int)
] = 8

Quantization bit-width used when embedding_compression is turboquant.

Must be one of the bit-widths supported by TurboQuant ({1, 2, 3, 4, 8}). Has no effect when embedding_compression is none.

embedding_compression_turboquant_chunk_size class-attribute instance-attribute

embedding_compression_turboquant_chunk_size: int = (
    pydantic.Field(default=4096, gt=0)
)

Chunk size used when embedding_compression is turboquant.

Embeddings are encoded in chunks of this size to bound temporary GPU memory usage. Has no effect unless embedding_compression is turboquant.

prompt_embeds_serialization_threads class-attribute instance-attribute

prompt_embeds_serialization_threads: PositiveInt = 8

Number of content parts each text SGT worker serializes concurrently for chat requests forwarded as prompt_embeds.

Each in-flight part holds a CPU copy of its embeddings and a torch.save staging buffer, so peak memory grows with this value. Each thread also keeps its staging buffer between requests, at the size of the largest part it has serialized, so a worker can retain up to this many times the largest part's serialized size. Set to 1 to serialize parts one at a time.

DefaultSamplingParams

Default sampling parameters for generation when not specified in the request.

Attributes:

Name Type Description
override_max_tokens int | None

Hard cap on max_tokens that takes precedence over any value in the request.

ignore_request_null_max_tokens bool

Ignore null values for max_tokens/max_completion_tokens in the request, and apply the next inferred value following precedence rules.

default_max_tokens int | None

When a request does not specify max_tokens or max_completion_tokens, this value will be injected into the request sent to the

allow_determining_default_max_tokens_from_upstream bool

Whether to try determining the max_tokens from the Upstream Provider.

allow_determining_default_max_tokens_from_sgt bool

Whether to try determining the max_tokens from the SGT's tokenizer.

override_max_tokens class-attribute instance-attribute

override_max_tokens: int | None = None

Hard cap on max_tokens that takes precedence over any value in the request.

This sits at the top of the precedence order and is applied even when the client explicitly provides a max_tokens or max_completion_tokens value, acting as a true operator override rather than a default.

None indicates that no override should be applied. -1 indicates that the overridden max tokens should be unlimited (i.e. "max_tokens" will have a value of null when serialized to json and sent to the upstream inference service).

ignore_request_null_max_tokens class-attribute instance-attribute

ignore_request_null_max_tokens: bool = True

Ignore null values for max_tokens/max_completion_tokens in the request, and apply the next inferred value following precedence rules.

The OpenAI API Spec reads an explicit null as "generate up to the remaining context window", but vLLM's /v1/completions endpoint coerces it to the field default of 16 tokens. Enabling this setting keeps the chat->completions inference path from forwarding a null that would be truncated that way, and instead falls through to the configured default or to the model's remaining context window.

On the chat->chat inference path the upstream already implements the OpenAI semantics, so this setting only decides whether the configured default_max_tokens may supersede an explicit client null.

default_max_tokens class-attribute instance-attribute

default_max_tokens: int | None = None

When a request does not specify max_tokens or max_completion_tokens, this value will be injected into the request sent to the upstream inference service.

None indicates that no default should be applied. -1 indicates that the default max tokens should be unlimited (i.e. "max_tokens" will have a value of null when serialized to json and sent to the upstream inference service).

allow_determining_default_max_tokens_from_upstream class-attribute instance-attribute

allow_determining_default_max_tokens_from_upstream: bool = (
    True
)

Whether to try determining the max_tokens from the Upstream Provider.

If none of the setting override, request, nor the default specify the max tokens value, allow determining the default max tokens value by querying the upstream provider's /v1/models endpoint. If this request is successful, and the provider exposes the model's maximum context window, the default max tokens for a request will be set to (model context window - input token count).

vLLM is the only tested upstream provider for this feature at the moment, but other providers may expose similar functionality.

allow_determining_default_max_tokens_from_sgt class-attribute instance-attribute

allow_determining_default_max_tokens_from_sgt: bool = True

Whether to try determining the max_tokens from the SGT's tokenizer.

If none of the setting override, request, nor the default specify the max tokens value, and determining default max tokens from upstream is disabled or fails, allow determining the default max tokens value based on the SGT's maximum context window. If this is enabled, and the SGT file specifies a maximum context window, the default max tokens for a request will be set to (SGT max context window - input token count). Note that this is usually an upper bound on the maximum content window of the hosted model. Upstream providers may host the model with a smaller context window.

BACKWARDS_COMPATIBLE_ENV_VARS module-attribute

BACKWARDS_COMPATIBLE_ENV_VARS: Final[Mapping[str, str]] = {
    "SGP_SGT_PATH": "SGP_SGT_TEXT__PATH",
    "SGP_SGT_NAME": "SGP_SGT_TEXT__NAME",
    "SGP_DEVICE": "SGP_SGT_TEXT__DEVICE",
    "SGP_NUM_SGT_WORKERS": "SGP_SGT_TEXT__NUM_WORKERS",
    "SGP_SGT_USE_CACHE": "SGP_SGT_TEXT__USE_CACHE",
    "SGP_TENSOR_PARALLEL_SIZE": "SGP_SGT_TEXT__TENSOR_PARALLEL_SIZE",
    "SGP_GRACE_PERIOD_SECONDS": "SGP_SGT_TEXT__GRACE_PERIOD_SECONDS",
    "SGP_WORKER_READY_TIMEOUT_SECONDS": "SGP_SGT_TEXT__WORKER_READY_TIMEOUT_SECONDS",
    "SGP_SGT_TORCH_DTYPE": "SGP_SGT_TEXT__TORCH_DTYPE",
    "SGP_SGT_NOISE_LAYER_ATTENTION": "SGP_SGT_TEXT__NOISE_LAYER_ATTENTION",
    "SGP_SGT_THIRD_PARTY_MODEL_PATH": "SGP_SGT_TEXT__THIRD_PARTY_MODEL_PATH",
    "SGP_SGT_TRUST_REMOTE_CODE": "SGP_SGT_TEXT__TRUST_REMOTE_CODE",
    "SGP_COMPILE_NOISE_LAYER_FORWARD": "SGP_SGT_TEXT__COMPILE_NOISE_LAYER_FORWARD",
    "SGP_RECONSTRUCTION_MAX_SEQUENCE_LENGTH": "SGP_RECONSTRUCTION_TEXT__MAX_SEQUENCE_LENGTH",
    "SGP_RECONSTRUCTION_MAX_NUM_EMBEDDINGS": "SGP_RECONSTRUCTION_TEXT__MAX_NUM_EMBEDDINGS",
    "SGP_RECONSTRUCTION_MAX_BATCH_SIZE": "SGP_RECONSTRUCTION_TEXT__MAX_BATCH_SIZE",
}

Mapping of backwards compatible environment variables to their new equivalents.

If an old environment variable is set and the new environment variable is not set, the value from the old environment variable will be used, and a warning will be logged.

If both the old and new environment variables are set, the value from the new environment variable will be used, a warning will be logged, and the old environment variable will be ignored.

If your deployment relies on any of the old environment variables, please update your deployment. The old environment variables are liable to be removed in a future major release.