inference/research
verifiedhigh · 0.90majorupdated 2026-10-03

Forcing the FlashInfer attention backend fails on head_dim 256/512 models

[major] kernel

Fields

id
gotchas/flashinfer-rejects-large-head-dim
Class
kernel
Severity
major
Concepts
blackwell-sm120gemma-4-31b-it
Symptom
Engine init aborts with: ValueError: Selected backend AttentionBackendEnum.FLASHINFER is not valid for this configuration. Reason: ['head_size not supported']. Reproduces for the bf16 base model, an FP8 checkpoint and an NVFP4 checkpoint of google/gemma-4-31b-it on an RTX PRO 6000 (SM120), whose config reports head_dim=256 and global_head_dim=512.
Root cause
FlashInfer's template instantiations do not cover these head sizes for this configuration, and vLLM's backend selection validates head_size before choosing a kernel. The override is accepted by the CLI and only fails at engine init.
Workaround
Drop the attention_config override and let vLLM choose; the same model loads and runs correctly under TRITON_ATTN. If you need FlashInfer specifically, check head_size support for your shape before selecting it.
notes
The reporter asks for either support or an earlier, clearer rejection at argument-parse time rather than an engine-init failure. Distinguish this from gotcha-sm120-sparse-mla-block-size, which is a different SM120 kernel-set gap (page_block_size, not head_size).

Notes

The reporter asks for either support or an earlier, clearer rejection at argument-parse time rather than an engine-init failure. Distinguish this from gotcha-sm120-sparse-mla-block-size, which is a different SM120 kernel-set gap (page_block_size, not head_size).

Cross-references