vLLM#56535: FlashInfer b12x MoE runs W4A16 checkpoints as W4A4
forum
Fields
id
sources/src5-gotcha-vllm-56535-flashinfer-b12x-w4a16Publisher
vLLM
Kind
forum
Published
2026-09-11
Accessed
2026-10-04
notes
FlashInfer flashinfer_b12x MoE backend runs W4A16 checkpoints as W4A4. Root cause: vLLM knows checkpoint is W4A16 but never tells FlashInfer. FlashInfer defaults to fp4 activation precision. Fix: pass quant_mode and activation_precision. GSM8K: 0.9037 (fix) vs 0.9136 (marlin). Demonstrates quantization accuracy loss from kernel misdispatch.
Notes
FlashInfer flashinfer_b12x MoE backend runs W4A16 checkpoints as W4A4. Root cause: vLLM knows checkpoint is W4A16 but never tells FlashInfer. FlashInfer defaults to fp4 activation precision. Fix: pass quant_mode and activation_precision. GSM8K: 0.9037 (fix) vs 0.9136 (marlin). Demonstrates quantization accuracy loss from kernel misdispatch.
Cross-references
This record references no other records.