AITER integration into SGLang for DeepSeek-R1 inference on MI300X

Fields

id

sources/amd-aiter-sglang-deepseek

URL

https://rocm.blogs.amd.com/artificial-intelligence/aiter-intergration-s/README.html

Publisher

AMD ROCm Blogs

Kind

benchmark

Published

2025-05-16

Accessed

2026-10-04

notes

Measured on MI300X with SGLang v0.4.4 + AITER. Replaces MoE (topk routing, sorting, blockscale FP8), FP8 blockscale GEMM, MLA decode/prefill attention, and all-reduce. DeepSeek-R1 total throughput: 14,635.98 token/s (median 2,255.85 token/s per decode step). Based on private repository.

Notes

Measured on MI300X with SGLang v0.4.4 + AITER. Replaces MoE (topk routing, sorting, blockscale FP8), FP8 blockscale GEMM, MLA decode/prefill attention, and all-reduce. DeepSeek-R1 total throughput: 14,635.98 token/s (median 2,255.85 token/s per decode step). Based on private repository.

Cross-references

This record references no other records.