inference/research
verifiedhigh · 0.95updated 2026-10-04

NVIDIA DGX B300, MLPerf Inference v6.1 Offline: Llama 2 70B on 8x B300-SXM-270GB, aggregate output throughput

109277 output tokens/s (aggregate, harness 'Tokens per second')

Fields

id
benchmarks/bench-mlperf-v6-1-nvidia-b300-llama2-70b-offline
Metric
tps_aggregate
Value
109,277
Unit
output tokens/s (aggregate, harness 'Tokens per second')
Model
Llama 2 70B (MLPerf v6.1 submission, FP4 weights)
Methodology
AGGREGATE. MLPerf Inference v6.1 Closed division, Datacenter suite, Offline scenario. LOADS: batch size NOT a fixed number and NOT PUBLISHED - the harness runs a closed-loop Offline queue and the submitter chooses the scheduler's internal batching, so there is no single batch size to state; the binding load conditions are instead (a) the benchmark's fixed input/output shape, (b) min_duration 600000 ms enforced by llama2-70b/user.conf, (c) min_query_count 2000, and (d) early stopping satisfied: Yes. Model shape: TOKENS_PER_SAMPLE 267.0 for this submission, so the figure is 267 x samples/s. Hardware: 8x NVIDIA B300-SXM-270GB, 1 node, aarch64 not stated for this entry but the DGX B300 platform is SXM; framework per summary.csv is 'TensorRT 10.14, CUDA 13.1, cuDNN 9.17, TensorRT-LLM feat/1.2-mlpinf, NVIDIA Dynamo mlperf-v6.0-dynamo-v0.8.0, vLLM CentML:mlperf-inf-mm-q3vl-v6.1'. POWER: has_power=False and design_power_watts=null - NO power measurement exists for this run and NO power limit is declared anywhere in the row. Samples per second 400.205. OMITTED BY SOURCE: TP size, KV-cache dtype, concurrency, clock state, cooling configuration.
Measured by
vendor
Reproducible
yes
notes
Read this together with bench-mlperf-v6-1-nvidia-b300-llama2-70b-server (106949.36 on the same node): the 2.2% gap is the ONLY thing separating them, and it is a scenario constraint, not a tuning difference. Two things a reader should NOT conclude from this number: it is not a per-GPU figure in any meaningful sense because the 8 GPUs are joined by NVLink and act as one 2.16TB memory domain, and it carries no watts, so it cannot be turned into a cost-per-token without an external measurement. vLLM appears in the framework string for this submission, but the LLM path is TensorRT-LLM 10.14 - do not read the vLLM string as the engine that produced the tokens. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b300). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

Read this together with bench-mlperf-v6-1-nvidia-b300-llama2-70b-server (106949.36 on the same node): the 2.2% gap is the ONLY thing separating them, and it is a scenario constraint, not a tuning difference. Two things a reader should NOT conclude from this number: it is not a per-GPU figure in any meaningful sense because the 8 GPUs are joined by NVLink and act as one 2.16TB memory domain, and it carries no watts, so it cannot be turned into a cost-per-token without an external measurement. vLLM appears in the framework string for this submission, but the LLM path is TensorRT-LLM 10.14 - do not read the vLLM string as the engine that produced the tokens. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-b300). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Format (1)

Interconnect (2)