AITER (AI Tensor Engine for ROCm) kernel library and measured speedups
Fields
flops/amd-rocm-aiter-kernelsNotes
AITER (AI Tensor Engine for ROCm) is AMD's high-performance AI operator library, providing optimized GPU kernels for inference and training. It supports multiple kernel backends: Triton, Composable Kernel (CK), and hand-tuned ASM. Measured speedups: (1) AITER block-scaled FP8 GEMM in vLLM 0.9.x: 19.4% request throughput increase, 18.6% output token throughput increase. (2) AITER MLA with matrix absorption on MI300X: up to 2x faster inference vs non-AITER, with speedup increasing with context length. (3) AITER Biased Group TopK for DeepSeek-V3: 4.9% request throughput, 6.1% TTFT reduction. (4) AITER integration in SGLang for DeepSeek-R1: 14,635.98 token/s total throughput. (5) ROCM_AITER_FA attention backend in vLLM: 2.7-4.4x higher TPS than ROCM_ATTN on MHA models. (6) ROCM_AITER_MLA: 1.2-1.5x higher TPS than TRITON_MLA on DeepSeek MLA. AITER is the default attention backend on ROCm in vLLM when VLLM_ROCM_USE_AITER=1 is set. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x, mi350x -> amd-instinct-mi350x, mi355x -> amd-instinct-mi355x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings (MI300X and MI325X are both CDNA3, MI350X/MI355X/MI350P all CDNA4) - the id is forced by the part name. This field is still a mixed bucket: it also carries the architecture codename values gfx942/gfx950, which are not ids and were left alone.