inference/research
verifiedhigh · 0.90updated 2026-10-04

vLLM#56535: FlashInfer b12x MoE runs W4A16 checkpoints as W4A4

forum

Fields

id
sources/src5-gotcha-vllm-56535-flashinfer-b12x-w4a16
Publisher
vLLM
Kind
forum
Published
2026-09-11
Accessed
2026-10-04
notes
FlashInfer flashinfer_b12x MoE backend runs W4A16 checkpoints as W4A4. Root cause: vLLM knows checkpoint is W4A16 but never tells FlashInfer. FlashInfer defaults to fp4 activation precision. Fix: pass quant_mode and activation_precision. GSM8K: 0.9037 (fix) vs 0.9136 (marlin). Demonstrates quantization accuracy loss from kernel misdispatch.

Notes

FlashInfer flashinfer_b12x MoE backend runs W4A16 checkpoints as W4A4. Root cause: vLLM knows checkpoint is W4A16 but never tells FlashInfer. FlashInfer defaults to fp4 activation precision. Fix: pass quant_mode and activation_precision. GSM8K: 0.9037 (fix) vs 0.9136 (marlin). Demonstrates quantization accuracy loss from kernel misdispatch.

Cross-references

This record references no other records.