inference/research
verifiedhigh · 0.95updated 2026-10-04

AMD 8xMI355X, MLPerf Inference v6.1 INTERACTIVE: Llama 2 70B aggregate output throughput while holding ttft<=450ms and tpot<=40ms

73191.51 output tokens/s (aggregate, harness 'Completed tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-amd-mi355x-llama2-70b-interactive
Metric
tps_aggregate
Value
73,191.51
Unit
output tokens/s (aggregate, harness 'Completed tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, INTERACTIVE scenario - a Server-scenario run with a much harder latency contract, and the third distinct performance regime on the same node. HARDWARE (MLPerf system id 8xMI355X_2xEPYC_9575F): 8xAMD Instinct MI355X 288GB HBM3e on 2x EPYC 9575F, 1 node. LOADS: the Interactive category requires loadgen to enforce ttft <= 450ms and tpot <= 40ms (MLCommons inference repo, language/llama2-70b/README.md, verbatim); on top of that it carries the Server arrival model, target_qps 0.5, min_duration 120000 ms, min_query_count 100. Batch size is not a fixed published number but is now bounded from above by the 40ms inter-token budget, which is what actually caps concurrency. Completed samples per second 258.22. Model shape TOKENS_PER_SAMPLE 267.0. POWER: has_power=False, design_power_watts=null, no declared cap. OMITTED BY SOURCE: TP size, KV dtype, achieved concurrency, clock state, cooling.
Measured by
vendor
Reproducible
yes
notes
READ THE SOURCE LOG BEFORE TRUSTING THE LABEL: the artifact at this path prints 'Scenario : Server' in its own header. The Interactive identity exists only in the directory name and the summary.csv Scenario column; the harness log gives no way to distinguish it from a Server run. THAT IS THE CENTRAL AUDITING TRAP IN MLPERF AND IT IS RECORDED AS data/gotchas/bench-mlperf-interactive-scenario-labels-itself-server.json. The substantive number: a 40ms inter-token cap costs 29% of throughput versus Offline on this node (106517 -> 73191.51). Anyone quoting an 'Interactive MLPerf number' as an achievable serving rate is quoting a number achieved while guaranteeing no user waits more than 40ms between tokens - which is a chatbot/voice interaction target, not a batch-serving target. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

READ THE SOURCE LOG BEFORE TRUSTING THE LABEL: the artifact at this path prints 'Scenario : Server' in its own header. The Interactive identity exists only in the directory name and the summary.csv Scenario column; the harness log gives no way to distinguish it from a Server run. THAT IS THE CENTRAL AUDITING TRAP IN MLPERF AND IT IS RECORDED AS data/gotchas/bench-mlperf-interactive-scenario-labels-itself-server.json. The substantive number: a 40ms inter-token cap costs 29% of throughput versus Offline on this node (106517 -> 73191.51). Anyone quoting an 'Interactive MLPerf number' as an achievable serving rate is quoting a number achieved while guaranteeing no user waits more than 40ms between tokens - which is a chatbot/voice interaction target, not a batch-serving target. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)