inference/research
verifiedhigh · 0.85updated 2026-10-03

MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out

15105 output tok/s (aggregate over all in-flight requests)

Fields

id
benchmarks/mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-output-throughput
Metric
tps_aggregate
Value
15,105
Unit
output tok/s (aggregate over all in-flight requests)
Model
Llama 3.1 70B
Methodology
AGGREGATE (not per-request) output token throughput under a closed-loop client fed requests at an infinite rate: 'no delay between messages', no latency constraint, saturation load. Model Llama 3.1 70B, FP8 weights. Tensor parallel TP=8 on both platforms. Input length 128 tokens, output length 2048 tokens, fixed. AMD side: 8x MI300X (192GB HBM3 each), EPYC host, vllm-dev:20250112 docker, benchmark harness per ROCm/MAD standalone-benchmarking procedure. NVIDIA side: 8x H100 SXM, Xeon host, trtllm-bench inside trtllm 0.16.0, following NVIDIA's own TensorRT-LLM perf-benchmarking doc. AMD states 'measurements were made with the same method as TRT-LLM published throughput data', i.e. method parity is asserted by AMD, not independently audited, and the two engines are NOT the same (vLLM vs TensorRT-LLM). OMITTED BY SOURCE: actual batch size / scheduler max_num_seqs, KV-cache dtype, GEMM tuning state, number of concurrent sequences resident, driver and ROCm/CUDA versions, clock/power limits. A local inference client on the host is a shared-resource risk on the EPYC side.
Measured by
vendor
Reproducible
no
notes
DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-h100 - same source table, same run conditions, only the silicon differs. Read the pair together. MI300X 15105 vs H100 15810 output tok/s = 0.96x, i.e. MI300X is ~4% behind on this shape. This is one of four rows for Llama 3.1 70B FP8 TP8 in the same table (128/4096 -> 0.92x, 500/2000 -> 0.87x, 2048/2048 -> 1.00x; geomean 0.94x). The pattern AMD highlights: MI300X does relatively better as output length grows, attributed to higher memory bandwidth shortening TPOT; it does worse at 500-token prompts. MI300X has 192GB HBM vs H100's 80GB, so the comparison is not memory-capacity-neutral - AMD argues the larger HBM lets the whole model plus KV cache sit on fewer GPUs and avoids AllReduce, which is a real confound when reading 'MI300X vs H100' as 'CDNA3 vs Hopper'. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-h100 - same source table, same run conditions, only the silicon differs. Read the pair together. MI300X 15105 vs H100 15810 output tok/s = 0.96x, i.e. MI300X is ~4% behind on this shape. This is one of four rows for Llama 3.1 70B FP8 TP8 in the same table (128/4096 -> 0.92x, 500/2000 -> 0.87x, 2048/2048 -> 1.00x; geomean 0.94x). The pattern AMD highlights: MI300X does relatively better as output length grows, attributed to higher memory bandwidth shortening TPOT; it does worse at 500-token prompts. MI300X has 192GB HBM vs H100's 80GB, so the comparison is not memory-capacity-neutral - AMD argues the larger HBM lets the whole model plus KV cache sit on fewer GPUs and avoids AllReduce, which is a real confound when reading 'MI300X vs H100' as 'CDNA3 vs Hopper'. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (2)

Engine (1)

Format (1)

Interconnect (5)