Skip to content

shm_tensor_arena

Enable vLLM's shared-memory tensor arena by default, and let deployments size it.

With tensor parallelism, vLLM's executor broadcasts every SchedulerOutput to each worker. Without the arena, a large CPU tensor such as a request's prompt_embeds goes out-of-band over a ZMQ socket, once per worker. The arena (--enable-shm-tensor-arena, vLLM 0.32) instead copies the tensor once into a shared-memory slot that every worker maps zero-copy, and page-locks, so the H2D copy is a direct DMA.

vLLM ships the arena disabled, with a fixed sizing (8 slots x 256 MiB) that it keeps off the env/CLI surface. A tensor larger than one slot silently takes the old path, and 256 MiB is only ~16k tokens of bf16 embeddings at hidden size 8192. This module:

  1. Flips the --enable-shm-tensor-arena argparse default to True. Opt out with --no-enable-shm-tensor-arena or SG_ENABLE_SHM_TENSOR_ARENA=0.
  2. Lets SG_SHM_TENSOR_ARENA_SLOT_GIB and SG_SHM_TENSOR_ARENA_SLOTS replace vLLM's sizing. Unset, vLLM's sizing is left untouched.

The sizing is read once, when the plugin loads, and applied in every vLLM process. Only the EngineCore, which builds the arena, needs it: workers adopt the writer's sizing from the arena's attach handle.

Warning

Under almost no circumstances should you need to import this module directly. If stainedglass_output_protection is installed, vLLM loads it automatically via the vllm.general_plugins entry point.

Functions:

Name Description
patch_shm_tensor_arena

Enable vLLM's shm tensor arena by default, sized by SG_SHM_TENSOR_ARENA_SLOT_GIB and SG_SHM_TENSOR_ARENA_SLOTS when set.

Attributes:

Name Type Description
SG_ENABLE_SHM_TENSOR_ARENA_ENV_VAR Final[str]

Set to 0 to leave vLLM's shm tensor arena, its default and its sizing, exactly as vLLM ships them.
SG_SHM_TENSOR_ARENA_SLOTS_ENV_VAR Final[str]

Number of arena slots, which bounds how many large tensors can be in flight on the fast path at once. Unset keeps vLLM's 8.
SG_SHM_TENSOR_ARENA_SLOT_GIB_ENV_VAR Final[str]

Size of one arena slot, in gibibytes. A tensor larger than one slot does not use the arena. Unset keeps vLLM's 256 MiB.

SG_ENABLE_SHM_TENSOR_ARENA_ENV_VAR module-attribute 

SG_ENABLE_SHM_TENSOR_ARENA_ENV_VAR: Final[str] = (
    "SG_ENABLE_SHM_TENSOR_ARENA"
)

Set to 0 to leave vLLM's shm tensor arena, its default and its sizing, exactly as vLLM ships them.

SG_SHM_TENSOR_ARENA_SLOTS_ENV_VAR module-attribute 

SG_SHM_TENSOR_ARENA_SLOTS_ENV_VAR: Final[str] = (
    "SG_SHM_TENSOR_ARENA_SLOTS"
)

Number of arena slots, which bounds how many large tensors can be in flight on the fast path at once. Unset keeps vLLM's 8.

SG_SHM_TENSOR_ARENA_SLOT_GIB_ENV_VAR module-attribute 

SG_SHM_TENSOR_ARENA_SLOT_GIB_ENV_VAR: Final[str] = (
    "SG_SHM_TENSOR_ARENA_SLOT_GIB"
)

Size of one arena slot, in gibibytes. A tensor larger than one slot does not use the arena. Unset keeps vLLM's 256 MiB.

patch_shm_tensor_arena 

patch_shm_tensor_arena() -> None

Enable vLLM's shm tensor arena by default, sized by SG_SHM_TENSOR_ARENA_SLOT_GIB and SG_SHM_TENSOR_ARENA_SLOTS when set.

Must run before the serve parser is built (for the default) and before the EngineCore builds its executor (for the sizing); vLLM's plugin loading does both. Only the CLI default moves: vllm.LLM(...) and other programmatic callers keep vLLM's own default. Idempotent, and a no-op when SG_ENABLE_SHM_TENSOR_ARENA=0.

Raises:

Type Description
ValueError

If SG_SHM_TENSOR_ARENA_SLOT_GIB or SG_SHM_TENSOR_ARENA_SLOTS is not a positive number.
RuntimeError

If the vLLM symbols this patch relies on are missing or changed.