inference/research
verifiedhigh · 0.95updated 2026-10-04

AMD 8xMI355X, MLPerf Inference v6.1 Offline: Llama 2 70B on AMD's own node, aggregate output throughput

106517 output tokens/s (aggregate, harness 'Tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-amd-mi355x-llama2-70b-offline
Metric
tps_aggregate
Value
106,517
Unit
output tokens/s (aggregate, harness 'Tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, Offline scenario, submitted by AMD itself on Supermicro AS-4126GS-NMR-LCC, 1 node, 8x AMD Instinct MI355X 288GB HBM3e. LOADS: batch size not a fixed published number (closed-loop Offline queue, submitter-chosen scheduler batching); binding conditions are min_duration 600000 ms, min_query_count 2000, early stopping satisfied: Yes. Model shape TOKENS_PER_SAMPLE 267.0. Framework per summary.csv: 'PyTorch 2.10.0+git8514f05, ROCm 7.2.2'; the LLM serving path for this submission is vLLM (the repo's other AMD v6.0/v6.1 records attribute MI355X Llama 2 70B runs to vLLM with matching vLLM version strings), so engine_id is recorded as vllm while noting the row itself names only PyTorch/ROCm. Samples per second 375.84. POWER: has_power=False, design_power_watts=null, no declared cap, no cooling configuration declared. OMITTED BY SOURCE: TP size, KV-cache dtype, concurrency, clock state.
Measured by
vendor
Reproducible
yes
notes
AMD's own v6.1 MI355X number lands within 2.6% of NVIDIA's DGX B300 on the same model and the same scenario (106517 vs 109277), despite B300 having 270GB against MI355X's 288GB. That near-tie is the most useful single fact in the v6.1 LLM set: on this model at FP4, an 8-GPU AMD node and an 8-GPU NVIDIA node are the same product. Do not upgrade this into a 'MI355X beats B300' claim - the 2.6% is well inside the spread between OEM implementations of the same GPU (Supermicro's MI355X node gives 102243, HPE's gives 102991, Oracle's gives 101553 for this exact scenario, a 1.4% band), and the FP4 formats are not even the same scheme (AMD mxfp4 vs NVIDIA nvfp4). INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

AMD's own v6.1 MI355X number lands within 2.6% of NVIDIA's DGX B300 on the same model and the same scenario (106517 vs 109277), despite B300 having 270GB against MI355X's 288GB. That near-tie is the most useful single fact in the v6.1 LLM set: on this model at FP4, an 8-GPU AMD node and an 8-GPU NVIDIA node are the same product. Do not upgrade this into a 'MI355X beats B300' claim - the 2.6% is well inside the spread between OEM implementations of the same GPU (Supermicro's MI355X node gives 102243, HPE's gives 102991, Oracle's gives 101553 for this exact scenario, a 1.4% band), and the FP4 formats are not even the same scheme (AMD mxfp4 vs NVIDIA nvfp4). INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)