inference/research
verifiedhigh · 0.95updated 2026-10-04

Cisco UCS C880A M8 8xH200-SXM at a DECLARED H200 TGP of 700W, MLPerf Inference v5.1 Offline: Llama 2 70B

34548.2 output tokens/s (aggregate, harness 'Tokens per second')

Fields

id
benchmarks/bench-mlperf-v5-1-cisco-h200-tgp700w-llama2-70b-offline
Metric
tps_aggregate
Value
34,548.2
Unit
output tokens/s (aggregate, harness 'Tokens per second')
Model
Llama 2 70B (MLPerf v5.1 submission, -99.9 accuracy track, FP8 weights)
Methodology
AGGREGATE. MLPerf Inference v5.1 Closed division, Datacenter suite, Offline scenario. LOADS: batch size not a fixed published number; llama2-70b/user.conf min_duration 600000 ms and min_query_count 2000. Hardware: 1 node, 8x NVIDIA H200-SXM-141GB, Cisco UCS C880A M8. Framework per v5.1 summary_results.json: 'TensorRT 10.11, CUDA 12.9'. POWER CONFIGURATION - THE POINT OF THIS RECORD: the v5.1 row's own free-text Notes field reads 'H200 TGP 700W'. That is a submitter-DECLARED per-accelerator thermal design power limit for this run, and it is not an MLPerf power measurement (has_power is false for this row; all 24 power-measured v5.1 rows are stable-diffusion, retinanet and dlrm, none is a Llama). H200-SXM's factory TDP is 700W, so this is the stock limit rather than a reduced one. Cooling configuration is not declared anywhere in the row. OMITTED BY SOURCE: TP size, KV dtype, concurrency, clock state.
Measured by
vendor
Reproducible
yes
notes
This is how a declared power limit actually reaches the MLCommons database: as free text in a Notes column, not as a structured field, and only in rounds up to v6.0. In v6.1 the Notes column is uniformly 'N/A' for every LLM row and design_power_watts is null for all 492 rows, so a v6.1 result carries NO power information of any kind. The consequence for anyone comparing across rounds: the v6.1 B300 result (109277 tok/s) and this v5.1 H200 result (34548.2 tok/s) differ by 3.2x, but only the older one has a stated power limit at all, and neither has a measured wattage. The comparison across those two rounds is not a clean hardware comparison and cannot be made into one from the published data. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-h200-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

This is how a declared power limit actually reaches the MLCommons database: as free text in a Notes column, not as a structured field, and only in rounds up to v6.0. In v6.1 the Notes column is uniformly 'N/A' for every LLM row and design_power_watts is null for all 492 rows, so a v6.1 result carries NO power information of any kind. The consequence for anyone comparing across rounds: the v6.1 B300 result (109277 tok/s) and this v5.1 H200 result (34548.2 tok/s) differ by 3.2x, but only the older one has a stated power limit at all, and neither has a measured wattage. The comparison across those two rounds is not a clean hardware comparison and cannot be made into one from the published data. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-h200-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (3)