MI300A x1 node (4 APU): Mistral-Small-24B-Base-2501, vLLM batch inference, TTFT 4.91 s (CUG 2025)
4910 ms time-to-first-token (mean over 100 prompts, whole node)
Fields
id
benchmarks/cug2025-mi300a-vllm-mistral-small-24b-ttft-msMetric
ttft_ms
Value
4,910
Unit
ms time-to-first-token (mean over 100 prompts, whole node)
Model
mistralai/Mistral-Small-24B-Base-2501 (24B dense teacher)
Methodology
THIRD-PARTY MEASURED, peer-reviewed (CUG 2025 proceedings, HLRS 'Hunter' HPE Cray EX4000). vLLM batch-inference mode on 1 MI300A node = 4 APUs (gfx942/CDNA3, 128GB unified HBM each, ~5.3 TB/s peak, Infinity Fabric/XGMI + PCIe Gen5 x16), ROCm 6.2.2, PyTorch 2.6.0. Workload = 100 diverse prompts; metrics are AGGREGATE across the batched run (prompts/sec and total tokens/sec), NOT per-request decode tok/s. Figure 5. No speculative decoding, no quantization beyond model-native precision.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
TTFT 4.91s on the dense 24B teacher. The 15B student's 50% TTFT reduction is the paper's headline inference result.
Notes
TTFT 4.91s on the dense 24B teacher. The 15B student's 50% TTFT reduction is the paper's headline inference result.