SGLang v0.5.18 (VibeSys-ported+tuned) on 4x MI300A: Qwen3.5-397B-A17B MXFP4 goodput 961 tok/s @48sess
961.0 tok/s goodput (aggregate over 48 concurrent sessions, SLO-bound)
Fields
id
benchmarks/syfi-mi300a-sglang-tuned-qwen35-397b-mxfp4-goodputMetric
tps_aggregate
Value
961
Unit
tok/s goodput (aggregate over 48 concurrent sessions, SLO-bound)
Model
Qwen3.5-397B-A17B
Methodology
THIRD-PARTY MEASURED (SyFI Lab, UW, blog 2026-10-08). 4x AMD Instinct MI300A (gfx942, 128GB HBM3 shared CPU+GPU per package, 512GB node). Model Qwen3.5-397B-A17B MXFP4 (amd/Qwen3.5-397B-A17B-MXFP4, ~17B active, 512 routed experts, hybrid 15 attn + 45 Gated-DeltaNet linear layers). Multi-turn chat benchmark, concurrency ramp 1->192 sessions, 3-6 turns x 80-300 gen tokens, full history resent each turn. GOODPUT = output tok/s for responses meeting SLO (p95 TPOT<=250ms, p95 TTFT<=10s). AGGREGATE over concurrent sessions. Baseline: SGLang v0.5.18 ported to gfx942 by VibeSys agents (fused HIP MXFP4 MoE kernel, pinned KV pool, pre-sharded weight loads, prebuilt AITER kernel cache), tuned with MTP speculative decoding + GEMM tuning tables. Peak at 48 concurrent sessions.
Measured by
third_party
Reproducible
yes
notes
Stock SGLang does NOT run this model on MI300A (AITER MXFP4 MoE kernels are gfx950-only); the 961 tok/s figure is a VibeSys-ported+tuned build, not upstream SGLang. Goodput is SLO-gated aggregate output throughput — reads lower than raw max-throughput.
Notes
Stock SGLang does NOT run this model on MI300A (AITER MXFP4 MoE kernels are gfx950-only); the 961 tok/s figure is a VibeSys-ported+tuned build, not upstream SGLang. Goodput is SLO-gated aggregate output throughput — reads lower than raw max-throughput.