vLLM issue #55131: Batch-invariant matmul is not actually batch-invariant, and TF32 causes precision degradation
forum
Fields
id
sources/nrg2-gh-vllm-55131-tf32-breaks-invariancePublisher
vLLM project
Kind
forum
Published
2026-09-03
Accessed
2026-10-03
notes
OPEN as of 2026-10-03. Reports that vLLM's vendored batch-invariant matmul is not batch invariant, and that on Ampere/Hopper tl.dot uses TF32 by default for fp32 inputs (same as torch.mm), which is wrong for precision-sensitive modules such as the DeepSeek Sparse Attention Indexer and the MoE router. Environment: torch 2.6.0, triton 3.2.0, H20e, driver 550.127.08, CUDA 12.4. Important for methodology: bit-exactness and numerical accuracy are DIFFERENT tests. A kernel can be batch-invariant and still lose accuracy through TF32; conversely a tolerance-based comparison can pass while output text still diverges. Test both.
Notes
OPEN as of 2026-10-03. Reports that vLLM's vendored batch-invariant matmul is not batch invariant, and that on Ampere/Hopper tl.dot uses TF32 by default for fp32 inputs (same as torch.mm), which is wrong for precision-sensitive modules such as the DeepSeek Sparse Attention Indexer and the MoE router. Environment: torch 2.6.0, triton 3.2.0, H20e, driver 550.127.08, CUDA 12.4. Important for methodology: bit-exactness and numerical accuracy are DIFFERENT tests. A kernel can be batch-invariant and still lose accuracy through TF32; conversely a tolerance-based comparison can pass while output text still diverges. Test both.
Cross-references
This record references no other records.