MI300A x1 node (4 APU): KafkaLM-15B, vLLM batch inference, 812 tok/s aggregate (CUG 2025)
812 tok/s aggregate across 100-prompt batch (whole node)
Fields
id
benchmarks/cug2025-mi300a-vllm-kafka15b-tok-sMetric
tps_aggregate
Value
812
Unit
tok/s aggregate across 100-prompt batch (whole node)
Model
Seedbox/KafkaLM-15B-Base (15B, 2:4-sparse distilled from Mistral-Small-24B)
Methodology
THIRD-PARTY MEASURED, peer-reviewed (CUG 2025 proceedings, HLRS 'Hunter' HPE Cray EX4000). vLLM batch-inference mode on 1 MI300A node = 4 APUs (gfx942/CDNA3, 128GB unified HBM each, ~5.3 TB/s peak, Infinity Fabric/XGMI + PCIe Gen5 x16), ROCm 6.2.2, PyTorch 2.6.0. Workload = 100 diverse prompts; metrics are AGGREGATE across the batched run (prompts/sec and total tokens/sec), NOT per-request decode tok/s. Figure 5. No speculative decoding, no quantization beyond model-native precision.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Aggregate decode+prefill throughput over the 100-prompt vLLM batch on a single 4-APU node. First published MI300A LLM inference figure. 6.55 prompts/s end-to-end. Confidence 0.72: peer-reviewed but single-run, throughput is node-aggregate not per-request.
Notes
Aggregate decode+prefill throughput over the 100-prompt vLLM batch on a single 4-APU node. First published MI300A LLM inference figure. 6.55 prompts/s end-to-end. Confidence 0.72: peer-reviewed but single-run, throughput is node-aggregate not per-request.