MI300A x1 node (4 APU): KafkaLM-15B, vLLM batch inference, TTFT 2.46 s (CUG 2025)
2460 ms time-to-first-token (mean over 100 prompts, whole node)
Fields
id
benchmarks/cug2025-mi300a-vllm-kafka15b-ttft-msMetric
ttft_ms
Value
2,460
Unit
ms time-to-first-token (mean over 100 prompts, whole node)
Model
Seedbox/KafkaLM-15B-Base (15B, 2:4-sparse distilled from Mistral-Small-24B)
Methodology
THIRD-PARTY MEASURED, peer-reviewed (CUG 2025 proceedings, HLRS 'Hunter' HPE Cray EX4000). vLLM batch-inference mode on 1 MI300A node = 4 APUs (gfx942/CDNA3, 128GB unified HBM each, ~5.3 TB/s peak, Infinity Fabric/XGMI + PCIe Gen5 x16), ROCm 6.2.2, PyTorch 2.6.0. Workload = 100 diverse prompts; metrics are AGGREGATE across the batched run (prompts/sec and total tokens/sec), NOT per-request decode tok/s. Figure 5. No speculative decoding, no quantization beyond model-native precision.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
TTFT 2.46s (p<0.01 vs teacher). Node-aggregate serving latency, not a single-stream number.
Notes
TTFT 2.46s (p<0.01 vs teacher). Node-aggregate serving latency, not a single-stream number.