inference/research
verifiedhigh · 0.90updated 2026-10-03

vLLM issue #55131: Batch-invariant matmul is not actually batch-invariant, and TF32 causes precision degradation

forum

Fields

id
sources/nrg2-gh-vllm-55131-tf32-breaks-invariance
Publisher
vLLM project
Kind
forum
Published
2026-09-03
Accessed
2026-10-03
notes
OPEN as of 2026-10-03. Reports that vLLM's vendored batch-invariant matmul is not batch invariant, and that on Ampere/Hopper tl.dot uses TF32 by default for fp32 inputs (same as torch.mm), which is wrong for precision-sensitive modules such as the DeepSeek Sparse Attention Indexer and the MoE router. Environment: torch 2.6.0, triton 3.2.0, H20e, driver 550.127.08, CUDA 12.4. Important for methodology: bit-exactness and numerical accuracy are DIFFERENT tests. A kernel can be batch-invariant and still lose accuracy through TF32; conversely a tolerance-based comparison can pass while output text still diverges. Test both.

Notes

OPEN as of 2026-10-03. Reports that vLLM's vendored batch-invariant matmul is not batch invariant, and that on Ampere/Hopper tl.dot uses TF32 by default for fp32 inputs (same as torch.mm), which is wrong for precision-sensitive modules such as the DeepSeek Sparse Attention Indexer and the MoE router. Environment: torch 2.6.0, triton 3.2.0, H20e, driver 550.127.08, CUDA 12.4. Important for methodology: bit-exactness and numerical accuracy are DIFFERENT tests. A kernel can be batch-invariant and still lose accuracy through TF32; conversely a tolerance-based comparison can pass while output text still diverges. Test both.

Cross-references

This record references no other records.