MLPerf v5.1 Jetson AGX Thor: llama3.1-8b-edge SingleStream TTFT p90 1397 ms
1396.97016 ms, 90th-percentile first-token latency, per query (SingleStream, concurrency 1)
Fields
id
benchmarks/mlperf-v5-1-thor-llama31-8b-edge-singlestream-ttftMetric
ttft_ms
Value
1,396.9702
Unit
ms, 90th-percentile first-token latency, per query (SingleStream, concurrency 1)
Model
llama3.1-8b-edge
Methodology
MLPerf Inference open-division submission, 1x NVIDIA Jetson AGX Thor Developer Kit 128GB (unified memory), TensorRT 10.8, CUDA 12.8. Quoted verbatim from performance/run_1/mlperf_log_summary.txt, Result is VALID. v5.1, llama3.1-8b-edge, FP4 weights, SingleStream, samples_per_query 1, max_async_queries 1, min_query_count 10000. '90.00 percentile first token latency (ns): 1396970160' = 1396.97 ms. Per request, not aggregate. The scenario also reports a 90th-percentile query latency of 4145.55 ms, which is end-to-end and is not stored.
Measured by
vendor
Reproducible
yes
notes
Open division. The log's early-stopping 90th-percentile estimate is 1397.97 ms; the recorded value is the measured 90th percentile, not the estimate.
Notes
Open division. The log's early-stopping 90th-percentile estimate is 1397.97 ms; the recorded value is the measured 90th percentile, not the estimate.