inference/research
verifiedhigh · 0.85updated 2026-10-03

MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8 batch 1: end-to-end time-to-last-token at 128-in/2048-out

19089 ms (time to last token, end-to-end, batch 1, per request)

Fields

id
benchmarks/mi300x-vllm-llama31-70b-fp8-tp8-ttlt-batch1
Metric
itl_ms
Value
19,089
Unit
ms (time to last token, end-to-end, batch 1, per request)
Model
Llama 3.1 70B
Methodology
PER-REQUEST latency, not aggregate. TTLT = total time from input submission to generation of all output tokens. Batch size 1 (single request in flight), TP=8, Llama 3.1 70B FP8, input 128 / output 2048. MI300X 19089 ms vs H100 20923 ms. The source gives the full batch sweep 1,2,4,8,16,32,64,128 for both hardware, plotted as throughput-vs-latency curves; only the batch-1 rows are reproduced as text here. OMITTED BY SOURCE: KV-cache dtype, scheduler settings, GEMM tuning, driver versions, and whether the client and server were co-located.
Measured by
vendor
Reproducible
no
notes
DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-ttlt-batch1-h100. At batch 1 MI300X is 1.10x BETTER on end-to-end latency (lower ms) despite being behind on aggregate throughput at high batch - which is the expected shape when decode is bandwidth-bound and MI300X has ~5 TB/s vs H100's ~3.35 TB/s HBM3. The latency advantage inverts by batch 64 (0.91x) and 128 (0.94x). metric is itl_ms because the schema has no end-to-end latency field; note the semantic mismatch: this is total request latency, NOT inter-token latency. Treat itl_ms in this record as 'time to last token'. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

DIRECTLY COMPARABLE to mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-ttlt-batch1-h100. At batch 1 MI300X is 1.10x BETTER on end-to-end latency (lower ms) despite being behind on aggregate throughput at high batch - which is the expected shape when decode is bandwidth-bound and MI300X has ~5 TB/s vs H100's ~3.35 TB/s HBM3. The latency advantage inverts by batch 64 (0.91x) and 128 (0.94x). metric is itl_ms because the schema has no end-to-end latency field; note the semantic mismatch: this is total request latency, NOT inter-token latency. Treat itl_ms in this record as 'time to last token'. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi300x, nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (2)

Engine (1)

Format (1)

Interconnect (5)