MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Server aggregate tokens/s
3181.65 tok/s AGGREGATE over 8x L40S 48GB, Server scenario
Fields
id
benchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-server-tokensMetric
tps_aggregate
Value
3,181.65
Unit
tok/s AGGREGATE over 8x L40S 48GB, Server scenario
Model
llama2-70b-99.9
Methodology
Same submission and stack as the Offline record. Server scenario, performance/run_1: Completed tokens per second 3181.65, Completed samples per second 10.90, Result is VALID, samples_per_query 1, target_qps 11, ttft_latency 2000 ms, tpot_latency 200 ms, both early-stopping checks passed. Quoted verbatim from mlperf_log_summary.txt. AGGREGATE across concurrent requests on all 8 accelerators, not per-request.
Measured by
vendor
Reproducible
yes
notes
Dell's 8x L40S Server result in the same round was 3201.13 tok/s, within 1 percent of this one, so the two boxes agree on Server and diverge on Offline. First-token latency from the same log: p50 198.7 ms, p99 525.5 ms, p99.9 711.7 ms, all inside the 2000 ms cap.
Notes
Dell's 8x L40S Server result in the same round was 3201.13 tok/s, within 1 percent of this one, so the two boxes agree on Server and diverge on Offline. First-token latency from the same log: p50 198.7 ms, p99 525.5 ms, p99.9 711.7 ms, all inside the 2000 ms cap.