vLLM auto-selects an FP8 linear kernel on RDNA4 that is slower than the one it picked before v0.28
[minor] framework
Fields
id
gotchas/vllm-rdna4-rowwise-fp8-kernel-auto-selectedClass
framework
Severity
minor
Symptom
On RDNA4 (Radeon AI PRO R9700, gfx1201), serving FP8 compressed-tensors checkpoints with --kv-cache-dtype fp8 and TRITON_ATTN, decode throughput depends on a kernel selector rather than on the hardware. v0.27.1 picks ChannelWiseTorchFP8ScaledMMLinear Kernel and gets 35.45 t/s on gemma-4-12B-it-FP8-Dynamic; v0.28.0 picks RowWiseTorchFP8ScaledMMLinearKernel and gets 32.22 t/s; v0.29.0 gets 26.16 t/s. The reporter's own v0.23.0 baseline was 33.8 t/s, so stock v0.28 is a 4.8% regression and v0.29 is 24% down. Prefill is unaffected (4116 tok/s).
Root cause
The capability gate on RowWiseTorchFP8ScaledMMLinearKernel changed. Before v0.28 it rejected RDNA4 because it required CDNA3 / compute capability 94+; from v0.28.0 the gate opens on gfx1201 and the slower kernel is selected over the better one. The reporter isolated this causally: removing a single list entry and changing nothing else recovers 10.0% in v0.28, with a run-to-run spread of 0.02-0.04 tok/s.
Workaround
Force the ChannelWise kernel until the fix lands (PR #57861, 'Avoid RowWise FP8 scaled MM on RDNA4'), or pin to v0.27.1 for FP8 serving on RDNA4.
notes
Scope is narrow and honestly stated: FP8 checkpoints on RDNA4 with the Triton attention backend. Severity is minor rather than major because prefill is untouched and the regression is 5-24% on decode only, and because the reporter flags the v0.29 numbers as weaker evidence (one spawn per arm) plus an unexplained second regression they could not isolate. The generalizable point for RDNA4: on a consumer-derived architecture vLLM kernel gates keyed to compute capability are the wrong abstraction, and auto-selection can get worse across versions with no configuration change on your side.
Notes
Scope is narrow and honestly stated: FP8 checkpoints on RDNA4 with the Triton attention backend. Severity is minor rather than major because prefill is untouched and the regression is 5-24% on decode only, and because the reporter flags the v0.29 numbers as weaker evidence (one spawn per arm) plus an unexplained second regression they could not isolate. The generalizable point for RDNA4: on a consumer-derived architecture vLLM kernel gates keyed to compute capability are the wrong abstraction, and auto-selection can get worse across versions with no configuration change on your side.