vLLM issue #57838: RowWiseTorchFP8ScaledMMLinearKernel auto-selected on RDNA4 costs 5-24% decode

Fields

id

sources/noncuda-vllm-57838-rdna4-rowwise-fp8

URL

https://github.com/vllm-project/vllm/issues/57838

Publisher

vllm-project/vllm

Kind

forum

Published

2026-09-20

Accessed

2026-10-03

notes

Radeon AI PRO R9700 (gfx1201), ROCm 7.2.3 / ROCm 10, FP8 compressed-tensors checkpoints with --kv-cache-dtype fp8 and TRITON_ATTN. v0.27.1 and earlier reject RowWiseTorchFP8ScaledMMLinearKernel on RDNA4 and pick ChannelWise (35.45 t/s); v0.28.0 starts selecting RowWise (32.22 t/s); v0.29.0 drops to 26.16 t/s. Prefill is unaffected (4116 tok/s on 0.29).

Notes

Radeon AI PRO R9700 (gfx1201), ROCm 7.2.3 / ROCm 10, FP8 compressed-tensors checkpoints with --kv-cache-dtype fp8 and TRITON_ATTN. v0.27.1 and earlier reject RowWiseTorchFP8ScaledMMLinearKernel on RDNA4 and pick ChannelWise (35.45 t/s); v0.28.0 starts selecting RowWise (32.22 t/s); v0.29.0 drops to 26.16 t/s. Prefill is unaffected (4116 tok/s on 0.29).

Cross-references

This record references no other records.