inference/research
verifiedmedium · 0.80updated 2026-10-04

rocBLAS/hipBLASLt tuning for MI300X/MI355X GEMM performance

compute bound

Fields

id
flops/amd-rocm-rocblas-tuning
Class
decode_gemm
Bound by
compute
Scales with
matrix_dimensionsprecisionbatch_size
Workarounds
hipBLASLt provides tuned GEMM kernels for AMD GPUs with architecture-specific optimizationsPYTORCH_TUNABLEOP_ENABLED=1 enables PyTorch's tunable GEMM operations on ROCmAITER provides alternative GEMM kernels that may outperform rocBLAS for specific shapes
notes
rocBLAS is AMD's BLAS library for ROCm, and hipBLASLt is its lightweight tuning API. For LLM inference, GEMM performance is critical for both prefill (large matrix multiplications) and decode (matrix-vector multiplications). The vLLM blog shows that enabling GEMM tuning (PYTORCH_TUNABLEOP_ENABLED=1) reduces latency by approximately 26% compared to the FP16 baseline (from 4.13s to 3.40s on Llama3-8B with FP8). AITER provides alternative GEMM kernels including block-scaled FP8 GEMM that can outperform rocBLAS for specific model architectures. The rocBLAS library is tuned for gfx942 (CDNA3) and gfx950 (CDNA4) with architecture-specific kernel selections. For MI300X, the key GEMM optimization is using the matrix cores via MFMA instructions for FP8/BF16/FP16 workloads. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x, mi350x -> amd-instinct-mi350x, mi355x -> amd-instinct-mi355x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings (MI300X and MI325X are both CDNA3, MI350X/MI355X/MI350P all CDNA4) - the id is forced by the part name. This field is still a mixed bucket: it also carries the architecture codename values gfx942/gfx950, which are not ids and were left alone.

Notes

rocBLAS is AMD's BLAS library for ROCm, and hipBLASLt is its lightweight tuning API. For LLM inference, GEMM performance is critical for both prefill (large matrix multiplications) and decode (matrix-vector multiplications). The vLLM blog shows that enabling GEMM tuning (PYTORCH_TUNABLEOP_ENABLED=1) reduces latency by approximately 26% compared to the FP16 baseline (from 4.13s to 3.40s on Llama3-8B with FP8). AITER provides alternative GEMM kernels including block-scaled FP8 GEMM that can outperform rocBLAS for specific model architectures. The rocBLAS library is tuned for gfx942 (CDNA3) and gfx950 (CDNA4) with architecture-specific kernel selections. For MI300X, the key GEMM optimization is using the matrix cores via MFMA instructions for FP8/BF16/FP16 workloads. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x, mi350x -> amd-instinct-mi350x, mi355x -> amd-instinct-mi355x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings (MI300X and MI325X are both CDNA3, MI350X/MI355X/MI350P all CDNA4) - the id is forced by the part name. This field is still a mixed bucket: it also carries the architecture codename values gfx942/gfx950, which are not ids and were left alone.

Cross-references