vLLM#55131: Batch-invariant matmul not actually batch-invariant, TF32 precision loss

Fields

id

sources/src5-gotcha-vllm-55131-batch-invariant-tf32

URL

https://github.com/vllm-project/vllm/issues/55131

Publisher

vLLM

Kind

forum

Published

2026-09-03

Accessed

2026-10-04

notes

Two bugs: (1) batch-invariant matmul returns different results for row subset vs full batch when K is power of 2 (512/1024/2048/4096). (2) fp32 path silently degrades to TF32 precision (tl.dot defaults to tf32). Affects precision-sensitive modules like DeepSeek Sparse Attention indexer. Fix: input_precision='ieee'.

Notes

Two bugs: (1) batch-invariant matmul returns different results for row subset vs full batch when K is power of 2 (512/1024/2048/4096). (2) fp32 path silently degrades to TF32 precision (tl.dot defaults to tf32). Affects precision-sensitive modules like DeepSeek Sparse Attention indexer. Fix: input_precision='ieee'.

Cross-references

This record references no other records.