inference/research
verifiedhigh · 0.95updated 2026-10-04

CoreWeave GB200 NVL72, MLPerf Inference v6.1 Server: Llama 2 70B across a full 72-GPU rack (18 nodes) under the Server latency constraint

910621.36 output tokens/s (aggregate across all 72 GPUs, harness 'Completed tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-coreweave-gb200-nvl72-llama2-70b-server
Metric
tps_aggregate
Value
910,621.36
Unit
output tokens/s (aggregate across all 72 GPUs, harness 'Completed tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, SERVER scenario at RACK scale. LOADS: target_qps 0.5 Poisson arrivals, min_duration 120000 ms, min_query_count 100, compiled-in per-query TTFT/TPOT 99th-percentile bounds, 'Performance constraints satisfied : Yes'. Batch size not a fixed published number. Completed samples per second 3353.29. Model shape TOKENS_PER_SAMPLE 267.0. POWER: has_power=False, design_power_watts=null, no declared cap, no cooling configuration. OMITTED BY SOURCE: replica/TP layout, fabric, KV dtype, concurrency, clock state.
Measured by
vendor
Reproducible
yes
notes
SERVER EXCEEDS OFFLINE ON THE SAME HARDWARE: 910621.36 Server versus 894002 Offline, +1.9%. This is the cleanest refutation of the folk rule that Offline is always the faster scenario. At 72-GPU scale the Server harness's closed-loop arrival pattern keeps all 18 nodes' schedulers saturated, whereas the Offline run amortises one enormous batch and pays its ramp and tail. The inversion is a fact about load shape, not about the hardware, and it means the scenario a vendor chooses to publish can flatter or penalise its own number by a couple of percent depending on scale. Any cross-vendor comparison must therefore name the scenario explicitly - see data/gotchas/bench-mlperf-offline-is-not-always-faster-than-server.json. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-gb200-nvl72). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

SERVER EXCEEDS OFFLINE ON THE SAME HARDWARE: 910621.36 Server versus 894002 Offline, +1.9%. This is the cleanest refutation of the folk rule that Offline is always the faster scenario. At 72-GPU scale the Server harness's closed-loop arrival pattern keeps all 18 nodes' schedulers saturated, whereas the Offline run amortises one enormous batch and pays its ramp and tail. The inversion is a fact about load shape, not about the hardware, and it means the scenario a vendor chooses to publish can flatter or penalise its own number by a couple of percent depending on scale. Any cross-vendor comparison must therefore name the scenario explicitly - see data/gotchas/bench-mlperf-offline-is-not-always-faster-than-server.json. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-gb200-nvl72). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)