inference/research
verifiedhigh · 0.93updated 2026-10-03

vLLM documentation: Batch Invariance (docs/features/batch_invariance.md, read from the vllm repo at main)

repo

Fields

id
sources/nrg2-vllm-docs-batch-invariance
Publisher
vLLM project
Kind
repo
Accessed
2026-10-03
notes
Official vLLM feature page, marked BETA: 'Some features are still under active development', tracking issue vllm-project/vllm#27433. States the guarantee batch invariance actually gives: 'the output of a model is deterministic and independent of the batch size or the order of requests in a batch.' Hardware support is explicitly bounded: NVIDIA GPUs of compute capability 8.0 or higher, and Intel XPUs with Triton support (Triton Attention backend required on XPU). So the guarantee is vendor-scoped - it is NOT claimed for AMD ROCm/HIP, and there is no Metal/TPU/CPU coverage. Upstream test suite exists at tests/v1/determinism/ with 10 files including test_batch_invariance.py, test_online_batch_invariance.py, test_cutlass_batch_invariant.py, test_nvfp4_batch_invariant.py, test_rms_norm_batch_invariant.py and test_xpu_batch_invariant_ut.py - i.e. the oracle is implemented as a per-op test suite, not a single end-to-end check.

Notes

Official vLLM feature page, marked BETA: 'Some features are still under active development', tracking issue vllm-project/vllm#27433. States the guarantee batch invariance actually gives: 'the output of a model is deterministic and independent of the batch size or the order of requests in a batch.' Hardware support is explicitly bounded: NVIDIA GPUs of compute capability 8.0 or higher, and Intel XPUs with Triton support (Triton Attention backend required on XPU). So the guarantee is vendor-scoped - it is NOT claimed for AMD ROCm/HIP, and there is no Metal/TPU/CPU coverage. Upstream test suite exists at tests/v1/determinism/ with 10 files including test_batch_invariance.py, test_online_batch_invariance.py, test_cutlass_batch_invariant.py, test_nvfp4_batch_invariant.py, test_rms_norm_batch_invariant.py and test_xpu_batch_invariant_ut.py - i.e. the oracle is implemented as a per-op test suite, not a single end-to-end check.

Cross-references

This record references no other records.