inference/research
verifiedmedium · 0.80updated 2026-10-04

Oracle 8xMI355X, MLPerf Inference v6.1 Server: Llama 3.1 8B - a case where Server throughput EXCEEDS Offline on one node

148594 output tokens/s (aggregate; summary.csv value, raw log not independently re-read for this record)

Fields

id
benchmarks/bench-mlperf-v6-1-oracle-mi355x-llama3-1-8b-server
Metric
tps_aggregate
Value
148,594
Unit
output tokens/s (aggregate; summary.csv value, raw log not independently re-read for this record)
Model
Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, SERVER scenario on Oracle's 8xMI355X_2xEPYC_9575F node. LOADS: target_qps 0.5 Poisson arrivals, min_duration 120000 ms, compiled-in per-query TTFT/TPOT 99th-percentile bounds; the row is marked compliance=closed with errors=0, i.e. it passed the submission checker, which does verify the Server latency constraints. Batch size not a fixed published number. Framework per summary.csv: 'vLLM 0.22.0, PyTorch 2.10.0, ROCm 7.2.2'. POWER: has_power=False, design_power_watts=null, no declared cap. OMITTED BY SOURCE: TP size, KV dtype, concurrency, clock state, cooling, and the raw harness log was not fetched for this record so the harness-precision value is not confirmed.
Measured by
vendor
Reproducible
yes
notes
RECORDED FOR AN ANOMALY, NOT FOR ITS VALUE. On this one node the Server figure (148594) is HIGHER than the Offline figure (144321) for the same model, the same 8 GPUs and the same quantization - a +3.0% inversion, whereas Dell's and HPE's MI355X nodes show Server below Offline by 6.2% and 6.1% respectively. Three near-identical AMD nodes, three different signs on the Offline-vs-Server relationship. This is the strongest available evidence that the Offline/Server ordering is an artifact of load shape and scheduling rather than a property of the hardware, and it is why this repository refuses to compare MLPerf numbers without the scenario named. Confidence is capped at 0.8 because the raw harness log was not independently re-read for this row, unlike every other benchmark record in this slice. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

RECORDED FOR AN ANOMALY, NOT FOR ITS VALUE. On this one node the Server figure (148594) is HIGHER than the Offline figure (144321) for the same model, the same 8 GPUs and the same quantization - a +3.0% inversion, whereas Dell's and HPE's MI355X nodes show Server below Offline by 6.2% and 6.1% respectively. Three near-identical AMD nodes, three different signs on the Offline-vs-Server relationship. This is the strongest available evidence that the Offline/Server ordering is an artifact of load shape and scheduling rather than a property of the hardware, and it is why this repository refuses to compare MLPerf numbers without the scenario named. Confidence is capped at 0.8 because the raw harness log was not independently re-read for this row, unlike every other benchmark record in this slice. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-instinct-mi355x). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)