inference/research
verifiedhigh · 0.85updated 2026-10-03

MI300X vs H100, vLLM, Llama 3.1 405B FP8 TP8: output token throughput at 128-in/4096-out (MI300X wins 1.62x)

3171 output tok/s (aggregate over all in-flight requests)

Fields

id
benchmarks/mi300x-vs-h100-vllm-llama31-405b-fp8-tp8-output-throughput
Metric
tps_aggregate
Value
3,171
Unit
output tok/s (aggregate over all in-flight requests)
Model
Llama 3.1 405B
Methodology
Same aggregate closed-loop, infinite-rate, no-latency-constraint setup as the 70B pair. Llama 3.1 405B FP8, TP=8, input 128 / output 4096. AMD: 8x MI300X + EPYC, vllm-dev:20250112. NVIDIA: 8x H100 SXM + Xeon, trtllm-bench trtllm 0.16.0. OMITTED BY SOURCE: batch size, max_num_seqs, KV-cache dtype, GEMM tuning state, driver versions. Note the 405B FP8 weights fit on 8xMI300X but AMD notes FP16 405B does not fit on a single H100 node - the comparison is done entirely in FP8 to make that moot.
Measured by
vendor
Reproducible
no
notes
DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-405b-fp8-tp8-h100. This is the most striking ROCm-beats-CUDA number in the whole space: 3171 vs 1957 output tok/s = 1.62x for MI300X. Across the four 405B shapes in the same table MI300X wins all four (1.25x, 1.62x, 1.13x, 1.28x; geomean 1.31x). So on 70B MI300X is roughly at parity or slightly behind (geomean 0.94x) while on 405B it is clearly ahead (geomean 1.31x) - the crossover is not about CDNA vs Hopper per se but about the bandwidth-to-capacity balance at 8-way TP for large weights. Vendor-published with the same engine-mismatch and memory-capacity confound caveats as the 70B pair. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-405b-fp8-tp8-h100. This is the most striking ROCm-beats-CUDA number in the whole space: 3171 vs 1957 output tok/s = 1.62x for MI300X. Across the four 405B shapes in the same table MI300X wins all four (1.25x, 1.62x, 1.13x, 1.28x; geomean 1.31x). So on 70B MI300X is roughly at parity or slightly behind (geomean 0.94x) while on 405B it is clearly ahead (geomean 1.31x) - the crossover is not about CDNA vs Hopper per se but about the bandwidth-to-capacity balance at 8-way TP for large weights. Vendor-published with the same engine-mismatch and memory-capacity confound caveats as the 70B pair. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (2)

Engine (1)

Format (1)

Interconnect (5)