MLPerf v6.0 Jetson AGX Thor: llama3.1-8b-edge SingleStream TPOT p90 9.81 ms
9.810269 ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)
Fields
id
benchmarks/mlperf-v6-0-thor-llama31-8b-edge-singlestream-itlMetric
itl_ms
Value
9.8103
Unit
ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)
Model
llama3.1-8b-edge
Methodology
MLPerf Inference open-division submission, 1x NVIDIA Jetson AGX Thor Developer Kit 128GB (unified memory), TensorRT 10.8, CUDA 12.8. Quoted verbatim from performance/run_1/mlperf_log_summary.txt, Result is VALID. v6.0, llama3.1-8b-edge, FP4 weights, SingleStream, concurrency 1. '90.00 percentile time to output token (ns): 9810269' = 9.81 ms. Per-token latency inside one request, not aggregate. Mean TPOT on the same log is 8.01 ms.
Measured by
vendor
Reproducible
yes
notes
Open division. v5.1's p90 TPOT for the same scenario was 21.64 ms; both are kept because the submissions do not document what changed.
Notes
Open division. v5.1's p90 TPOT for the same scenario was 21.64 ms; both are kept because the submissions do not document what changed.