inference/research
verifiedhigh · 0.95updated 2026-10-04

Dell XE9785L 8xMI355X, MLPerf Inference v6.1 INTERACTIVE: Llama 3.1 8B while holding an inter-token latency cap

130539.57 output tokens/s (aggregate, harness 'Completed tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-dell-mi355x-llama3-1-8b-interactive
Metric
tps_aggregate
Value
130,539.57
Unit
output tokens/s (aggregate, harness 'Completed tokens per second')
Model
Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, INTERACTIVE scenario, stored under a directory named Interactive and published in a Scenario column reading Interactive. HARDWARE (MLPerf system id XE9785L_8xMI355X): 8xAMD Instinct MI355X 288GB HBM3e, 1 node. LOADS: Server arrival model (target_qps 0.5, min_duration 120000 ms) PLUS the Interactive category's hard inter-token bound; the MLCommons reference repo states the Interactive latency contract as ttft <= 450ms and tpot <= 40ms for this benchmark family. Batch size not a fixed published number, but bounded from above by the inter-token budget. Completed samples per second 1019.06. Model shape GEN_LEN 8127480. POWER: has_power=False, design_power_watts=null, no declared cap. OMITTED BY SOURCE: TP size, KV dtype, achieved concurrency, clock state, cooling.
Measured by
vendor
Reproducible
yes
notes
AUDIT WARNING ON THIS RECORD: the harness log at this path prints 'Scenario : Server' as its own header, and it reports 'Performance constraints satisfied : Yes' exactly as a Server run does. Nothing in the raw artifact identifies it as Interactive. The directory name and the summary.csv column are the only evidence. Anyone auditing MLPerf by reading raw logs - the most defensible way to audit - will silently misclassify every Interactive result in the database as a Server result. That defect is recorded as data/gotchas/bench-mlperf-interactive-scenario-labels-itself-server.json. Substantively: 130540 tok/s is 17.6% below Offline and 12.1% below Server on this same node, which is the cost of guaranteeing no user waits more than 40ms between tokens. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

AUDIT WARNING ON THIS RECORD: the harness log at this path prints 'Scenario : Server' as its own header, and it reports 'Performance constraints satisfied : Yes' exactly as a Server run does. Nothing in the raw artifact identifies it as Interactive. The directory name and the summary.csv column are the only evidence. Anyone auditing MLPerf by reading raw logs - the most defensible way to audit - will silently misclassify every Interactive result in the database as a Server result. That defect is recorded as data/gotchas/bench-mlperf-interactive-scenario-labels-itself-server.json. Substantively: 130540 tok/s is 17.6% below Offline and 12.1% below Server on this same node, which is the cost of guaranteeing no user waits more than 40ms between tokens. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)