MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Offline aggregate tokens/s
3655.89 tok/s AGGREGATE over 8x L40S 48GB
Fields
id
benchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-offline-tokensMetric
tps_aggregate
Value
3,655.89
Unit
tok/s AGGREGATE over 8x L40S 48GB
Model
llama2-70b-99.9
Methodology
MLPerf Inference v5.0 closed division, datacenter suite. HPE ProLiant Compute DL380a Gen12, 8x NVIDIA L40S 48GB behind a PCIe Gen5 switch, 2x Xeon 6740E. The submission sw_notes say the Llama2-70b workload used the MLPerf v4.1 TensorRT-LLM stack, with CUDA 12.8 and driver 570.86.15. Offline scenario, performance/run_1: Tokens per second 3655.89, Result is VALID, samples_per_query 31680, target_qps 12, ttft_latency 2000 ms, tpot_latency 200 ms. Quoted verbatim from mlperf_log_summary.txt. AGGREGATE across all 8 accelerators, not per-request.
Measured by
vendor
Reproducible
yes
notes
Second independent 8x L40S measurement in the same round: Dell XE7745 scored 3481.53 tok/s Offline on the same model (mlperf-v5-0-dell-xe7745-8xl40s-llama2-70b-offline-tokens), about 5 percent lower. Dividing this aggregate by 8 gives roughly 457 tok/s per card, which is not a per-request decode rate. The 99 and 99.9 accuracy targets printed identical performance numbers.
Notes
Second independent 8x L40S measurement in the same round: Dell XE7745 scored 3481.53 tok/s Offline on the same model (mlperf-v5-0-dell-xe7745-8xl40s-llama2-70b-offline-tokens), about 5 percent lower. Dividing this aggregate by 8 gives roughly 457 tok/s per card, which is not a per-request decode rate. The 99 and 99.9 accuracy targets printed identical performance numbers.