MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Server p99 first-token latency
525.5 ms, p99 first-token latency at an 11 qps target, 8x L40S
Fields
id
benchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-server-ttft-p99Metric
ttft_ms
Value
525.5
Unit
ms, p99 first-token latency at an 11 qps target, 8x L40S
Model
llama2-70b-99.9
Methodology
Same Server run as mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-server-tokens. mlperf_log_summary.txt prints 99.00 percentile first token latency 525500719 ns, which is 525.5 ms. This is the latency of one request inside an aggregate serving run (samples_per_query 1, target_qps 11, TTFT cap 2000 ms), not a single-stream batch-1 measurement. p50 was 198.7 ms and p99.9 was 711.7 ms in the same log.
Measured by
vendor
Reproducible
yes
notes
Recorded at p99 rather than the mean because the Server scenario's constraint is a percentile cap. Not comparable with a batch-1 TTFT from llama-bench.
Notes
Recorded at p99 rather than the mean because the Server scenario's constraint is a percentile cap. Not comparable with a batch-1 TTFT from llama-bench.