inference/research
verifiedhigh · 0.95updated 2026-10-04

NVIDIA GB300 NVL72 8-GPU slice (2 nodes), MLPerf Inference v6.1 Offline: Llama 2 70B - the per-GPU-efficient end of the same rack platform

131464 output tokens/s (aggregate across 8 GPUs on 2 nodes, harness 'Tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-nvidia-gb300-8gpu-llama2-70b-offline
Metric
tps_aggregate
Value
131,464
Unit
output tokens/s (aggregate across 8 GPUs on 2 nodes, harness 'Tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, Offline scenario on an 8-GPU SLICE of a GB300 NVL72 (72 GPUs total on the NVL72), taken across 2 nodes with 4 accelerators per node. LOADS: batch size not a fixed published number; min_duration 600000 ms, min_query_count 2000, early stopping satisfied: Yes. Model shape TOKENS_PER_SAMPLE 267.0. Framework per summary.csv: 'TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML:mlperf-inf-mm-q3vl-v6.1'. Samples per second 481.461. POWER: has_power=False, design_power_watts=null, no declared cap, no cooling configuration. OMITTED BY SOURCE: TP size, KV dtype, concurrency, clock state.
Measured by
vendor
Reproducible
yes
notes
THE DECISION-RELEVANT NUMBER IN THE v6.1 SET, AND IT POINTS THE OPPOSITE WAY FROM THE HEADLINE. NVIDIA's own 8-GPU slice of a GB300 NVL72 delivers 16433 tok/s per GPU; CoreWeave's full 72-GPU rack of the same platform delivers 15779 tok/s per GPU. Nine times the GPUs for 8.6x the throughput. The NVL72's scale-up fabric buys almost nothing at Llama 2 70B FP4 because a single 8-GPU node already holds the entire quantized model and is not bandwidth-starved on weights. Note also that this 8-GPU slice is 20% FASTER per GPU than the 8-GPU B300 SXM node (13660/GPU, 109277 tok/s) despite both having 270GB and the same TensorRT-LLM 10.14 stack - GB300's 288GB and newer memory are worth having, but a rack of them is not worth having for this workload. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b300). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

THE DECISION-RELEVANT NUMBER IN THE v6.1 SET, AND IT POINTS THE OPPOSITE WAY FROM THE HEADLINE. NVIDIA's own 8-GPU slice of a GB300 NVL72 delivers 16433 tok/s per GPU; CoreWeave's full 72-GPU rack of the same platform delivers 15779 tok/s per GPU. Nine times the GPUs for 8.6x the throughput. The NVL72's scale-up fabric buys almost nothing at Llama 2 70B FP4 because a single 8-GPU node already holds the entire quantized model and is not bandwidth-starved on weights. Note also that this 8-GPU slice is 20% FASTER per GPU than the 8-GPU B300 SXM node (13660/GPU, 109277 tok/s) despite both having 270GB and the same TensorRT-LLM 10.14 stack - GB300's 288GB and newer memory are worth having, but a rack of them is not worth having for this workload. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b300). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)