inference/research
verifiedhigh · 0.85updated 2026-10-03

H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out

15810 output tok/s (aggregate over all in-flight requests)

Fields

id
benchmarks/mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-h100
Metric
tps_aggregate
Value
15,810
Unit
output tok/s (aggregate over all in-flight requests)
Model
Llama 3.1 70B
Methodology
NVIDIA half of a head-to-head published by AMD. Same aggregate closed-loop, infinite-rate, no-latency-constraint setup as the paired MI300X record. Llama 3.1 70B FP8, TP=8, input 128 / output 2048. 8x H100 SXM on a Xeon host, measured with trtllm-bench in trtllm 0.16.0 following NVIDIA's published TensorRT-LLM perf-benchmarking method. Engine field is set to vllm because that is the engine of the comparison; the NVIDIA number itself was produced by TensorRT-LLM, not vLLM - this asymmetry is the single biggest caveat on the pair. OMITTED BY SOURCE: batch size, max_num_seqs, KV-cache dtype, GEMM tuning state, TensorRT-LLM engine flags, CUDA/driver version, power limits.
Measured by
vendor
Reproducible
no
notes
DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-output-throughput. IMPORTANT HONESTY CAVEAT: this is NVIDIA data measured by AMD, not by NVIDIA. NVIDIA does not publish a matching 128/2048 Llama 3.1 70B FP8 TP8 vLLM figure in this source. Also engine-mismatched (TensorRT-LLM on NVIDIA, vLLM on AMD), so the delta blends silicon and software. It should be read as 'AMD's best effort at parity under its own harness', not as an independent CUDA benchmark. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-output-throughput. IMPORTANT HONESTY CAVEAT: this is NVIDIA data measured by AMD, not by NVIDIA. NVIDIA does not publish a matching 128/2048 Llama 3.1 70B FP8 TP8 vLLM figure in this source. Also engine-mismatched (TensorRT-LLM on NVIDIA, vLLM on AMD), so the delta blends silicon and software. It should be read as 'AMD's best effort at parity under its own harness', not as an independent CUDA benchmark. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)