MI300A x1 node (4 APU): Mistral-Small-24B-Base-2501, vLLM batch inference, 579 tok/s aggregate (CUG 2025)
579 tok/s aggregate across 100-prompt batch (whole node)
Fields
id
benchmarks/cug2025-mi300a-vllm-mistral-small-24b-tok-sMetric
tps_aggregate
Value
579
Unit
tok/s aggregate across 100-prompt batch (whole node)
Model
mistralai/Mistral-Small-24B-Base-2501 (24B dense teacher)
Methodology
THIRD-PARTY MEASURED, peer-reviewed (CUG 2025 proceedings, HLRS 'Hunter' HPE Cray EX4000). vLLM batch-inference mode on 1 MI300A node = 4 APUs (gfx942/CDNA3, 128GB unified HBM each, ~5.3 TB/s peak, Infinity Fabric/XGMI + PCIe Gen5 x16), ROCm 6.2.2, PyTorch 2.6.0. Workload = 100 diverse prompts; metrics are AGGREGATE across the batched run (prompts/sec and total tokens/sec), NOT per-request decode tok/s. Figure 5. No speculative decoding, no quantization beyond model-native precision.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Aggregate decode+prefill throughput over the 100-prompt vLLM batch on a single 4-APU node. 4.7 prompts/s end-to-end. Confidence 0.72: peer-reviewed but single-run, node-aggregate not per-request; the 24B dense teacher is the baseline against which the 15B student's 40% throughput gain is measured.
Notes
Aggregate decode+prefill throughput over the 100-prompt vLLM batch on a single 4-APU node. 4.7 prompts/s end-to-end. Confidence 0.72: peer-reviewed but single-run, node-aggregate not per-request; the 24B dense teacher is the baseline against which the 15B student's 40% throughput gain is measured.