inference/research
verifiedhigh · 0.95updated 2026-10-04

CoreWeave GB200 NVL72, MLPerf Inference v6.1 Offline: Llama 2 70B across a full 72-GPU rack (18 nodes)

894002 output tokens/s (aggregate across all 72 GPUs, harness 'Tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-coreweave-gb200-nvl72-llama2-70b-offline
Metric
tps_aggregate
Value
894,002
Unit
output tokens/s (aggregate across all 72 GPUs, harness 'Tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, Offline scenario at RACK scale. LOADS: batch size not a fixed published number; min_duration 600000 ms, min_query_count 2000, early stopping satisfied: Yes. Model shape TOKENS_PER_SAMPLE 267.0. Hardware: 72x NVIDIA GB200-186GB aarch64, 18 nodes, 4 accelerators per node - one whole GB200 NVL72. Framework per summary.csv: 'TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0'. Samples per second 3292.1. POWER: has_power=False, design_power_watts=null - a full NVL72 rack result with no watts anywhere in the row, which is remarkable given the rack's power draw is a primary argument for the platform. OMITTED BY SOURCE: replica count vs TP/EP layout, network fabric and topology, KV dtype, concurrency, clock state, cooling (an NVL72 is direct-liquid in practice but the row does not say so).
Measured by
vendor
Reproducible
yes
notes
Per-GPU this is 12417 tok/s, well below NVIDIA's own 8-GPU slice of the same-generation GB300 (16433/GPU, see bench-mlperf-v6-1-nvidia-gb300-8gpu-llama2-70b-offline) and below the single-node B300 (13660/GPU). Cross-referentially: AMD's own 11-node 88xMI355X v6.0 run reached 1042110 tok/s Offline, i.e. 11842/GPU. Neither rack result is evidence that scale-up fabric pays off at this model size - both are per-GPU losses relative to an 8-GPU node, because Llama 2 70B in FP4 is small enough that a single node already holds the whole model and the extra machines add communication and scheduling rather than removing a bottleneck. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-gb200-nvl72). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

Per-GPU this is 12417 tok/s, well below NVIDIA's own 8-GPU slice of the same-generation GB300 (16433/GPU, see bench-mlperf-v6-1-nvidia-gb300-8gpu-llama2-70b-offline) and below the single-node B300 (13660/GPU). Cross-referentially: AMD's own 11-node 88xMI355X v6.0 run reached 1042110 tok/s Offline, i.e. 11842/GPU. Neither rack result is evidence that scale-up fabric pays off at this model size - both are per-GPU losses relative to an 8-GPU node, because Llama 2 70B in FP4 is small enough that a single node already holds the whole model and the extra machines add communication and scheduling rather than removing a bottleneck. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-gb200-nvl72). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)