inference/research
verifiedhigh · 0.90updated 2026-10-04

vLLM ROCm-specific flags, environment variables, and known-good configuration

launch_overhead bound

Fields

id
flops/amd-vllm-rocm-flags
Class
custom_kernel
Bound by
launch_overhead
Scales with
model_architectureprecisiontensor_parallel_size
Workarounds
VLLM_ROCM_USE_AITER=1: enables AITER as default kernel backend (attention, MoE, GEMM)VLLM_ROCM_USE_AITER_LINEAR=1: enables AITER block-scaled FP8 GEMM (required for DeepSeek models)VLLM_ROCM_USE_AITER_MHA=1: enables AITER multi-head attentionVLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4: enables QuickReduce with INT4 quantizationVLLM_ROCM_USE_AITER_TRITON_FUSED_ROPE_ZEROS_KV_CACHE=1: enables fused RoPE + zeros KV cacheVLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1: enables AMD's preshuffled KV cache layout for 15-20% decode throughput improvementPYTORCH_TUNABLEOP_ENABLED=1: enables PyTorch tunable GEMM operations (~26% latency reduction)--attention-backend ROCM_AITER_FA: 3-path routing (prefill/extend/decode) with AITER primitives--attention-backend ROCM_AITER_MLA: AITER MLA backend for DeepSeek models--kv-cache-dtype fp8: FP8 KV cache to halve KV memory usage--compilation-config '{"custom_ops": ["-rms_norm", "-quant_fp8"], "cudagraph_mode": "FULL_AND_PIECEWISE"}': graph capture + custom ops
notes
vLLM on ROCm has extensive AMD-specific configuration. Key environment variables: VLLM_ROCM_USE_AITER=1 (master switch for AITER backends), VLLM_ROCM_USE_AITER_LINEAR=1 (block-scaled FP8 GEMM for DeepSeek), VLLM_ROCM_USE_AITER_MHA=1 (MHA attention), VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 (QuickReduce communication), VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (shuffled KV layout for 15-20% decode improvement). Key flags: --attention-backend ROCM_AITER_FA (3-path routing, 2.7-4.4x TPS vs ROCM_ATTN), --attention-backend ROCM_AITER_MLA (1.2-1.5x TPS vs TRITON_MLA), --kv-cache-dtype fp8, --compilation-config with cudagraph_mode FULL_AND_PIECEWISE. Known-good configuration for MI355X (from AMD's blog): VLLM_ROCM_USE_AITER=1, VLLM_ROCM_USE_AITER_MHA=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4, --tensor-parallel-size 8, --max-num-batched-tokens 8192, --kv-cache-dtype fp8, --compilation-config with custom_ops and cudagraph_mode. For MI300X: PYTORCH_TUNABLEOP_ENABLED=1 gives ~26% latency reduction. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x, mi350x -> amd-instinct-mi350x, mi355x -> amd-instinct-mi355x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings (MI300X and MI325X are both CDNA3, MI350X/MI355X/MI350P all CDNA4) - the id is forced by the part name. This field is still a mixed bucket: it also carries the architecture codename values gfx942/gfx950, which are not ids and were left alone.

Notes

vLLM on ROCm has extensive AMD-specific configuration. Key environment variables: VLLM_ROCM_USE_AITER=1 (master switch for AITER backends), VLLM_ROCM_USE_AITER_LINEAR=1 (block-scaled FP8 GEMM for DeepSeek), VLLM_ROCM_USE_AITER_MHA=1 (MHA attention), VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 (QuickReduce communication), VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 (shuffled KV layout for 15-20% decode improvement). Key flags: --attention-backend ROCM_AITER_FA (3-path routing, 2.7-4.4x TPS vs ROCM_ATTN), --attention-backend ROCM_AITER_MLA (1.2-1.5x TPS vs TRITON_MLA), --kv-cache-dtype fp8, --compilation-config with cudagraph_mode FULL_AND_PIECEWISE. Known-good configuration for MI355X (from AMD's blog): VLLM_ROCM_USE_AITER=1, VLLM_ROCM_USE_AITER_MHA=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4, --tensor-parallel-size 8, --max-num-batched-tokens 8192, --kv-cache-dtype fp8, --compilation-config with custom_ops and cudagraph_mode. For MI300X: PYTORCH_TUNABLEOP_ENABLED=1 gives ~26% latency reduction. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x, mi350x -> amd-instinct-mi350x, mi355x -> amd-instinct-mi355x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings (MI300X and MI325X are both CDNA3, MI350X/MI355X/MI350P all CDNA4) - the id is forced by the part name. This field is still a mixed bucket: it also carries the architecture codename values gfx942/gfx950, which are not ids and were left alone.

Cross-references

Hardware-dependent (4)