MLPerf v5.0 Jetson AGX Thor: llama2-70b-99 Offline 573.765 tok/s aggregate
573.765 tok/s AGGREGATE, single Thor module, Offline scenario
Fields
id
benchmarks/mlperf-v5-0-thor-llama2-70b-offline-tokensMetric
tps_aggregate
Value
573.765
Unit
tok/s AGGREGATE, single Thor module, Offline scenario
Model
llama2-70b-99
Methodology
MLPerf Inference open-division submission, 1x NVIDIA Jetson AGX Thor Developer Kit 128GB (unified memory), TensorRT 10.8, CUDA 12.8. Quoted verbatim from performance/run_1/mlperf_log_summary.txt, Result is VALID. v5.0, llama2-70b-99, FP16 weights, Offline. samples_per_query 24576, target_qps 1.93, min_duration 2400000 ms. 'Tokens per second: 573.765' (2.11134 samples/s). Offline is a throughput scenario: the number is aggregate system throughput, not a per-request decode rate. The llama2-70b-99.9 submission reports the identical performance log, so it is not a second record.
Measured by
vendor
Reproducible
yes
notes
Open division, not closed. FP16 70B on 128 GB unified memory. Per-request decode speed is not reported by the Offline scenario.
Notes
Open division, not closed. FP16 70B on 128 GB unified memory. Per-request decode speed is not reported by the Offline scenario.