MLPerf v5.1 Jetson AGX Thor: llama3.1-8b-edge SingleStream TPOT p90 21.64 ms
21.641875 ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)
Fields
id
benchmarks/mlperf-v5-1-thor-llama31-8b-edge-singlestream-itlMetric
itl_ms
Value
21.6419
Unit
ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)
Model
llama3.1-8b-edge
Methodology
MLPerf Inference open-division submission, 1x NVIDIA Jetson AGX Thor Developer Kit 128GB (unified memory), TensorRT 10.8, CUDA 12.8. Quoted verbatim from performance/run_1/mlperf_log_summary.txt, Result is VALID. v5.1, llama3.1-8b-edge, FP4 weights, SingleStream, concurrency 1. '90.00 percentile time to output token (ns): 21641875' = 21.64 ms. This is the per-token generation latency inside one request, not an aggregate across requests. Mean TPOT on the same log is 21.29 ms.
Measured by
vendor
Reproducible
yes
notes
Open division. Inverse of the p90 is about 46 tok/s per request, but that inverse is not what was measured and is not stored.
Notes
Open division. Inverse of the p90 is about 46 tok/s per request, but that inverse is not what was measured and is not stored.