Jetson AGX Thor llama.cpp: Qwen2.5-1.5B Q4_K_M generate 128 (Multimodalflow)
106.8 tok/s (llama-bench tg, per-request)
Fields
id
benchmarks/thor-llamacpp-qwen25-15b-q4km-tg128Metric
decode_tok_s
Value
106.8
Unit
tok/s (llama-bench tg, per-request)
Model
Qwen2.5-1.5B-Instruct Q4_K_M
Methodology
Multimodalflow live-hardware post dated 2026-06-01 on a Jetson AGX Thor Developer Kit. CUDA 13.0, compute capability 11.0, unified memory reported as 125,771 MiB (~123 GB), kernel 6.8.12-tegra. Not the MLPerf agentic-edge submission already in this repo. llama.cpp llama-bench, CUDA backend, flash attention on (`-fa 1`), all layers on GPU (`-ngl 999`). Model file qwen2.5-1.5b-instruct-q4_k_m.gguf, 1.04 GiB. Command: llama-bench -p 128,512 -n 128,256 -r 3. This record is generate 128 tokens: 106.8 +/- 6.4 t/s. Generate 256 measured 112.8 +/- 0.1 t/s and is noted, not stored separately. llama.cpp commit is not printed. Per request, batch 1.
Measured by
third_party
Reproducible
yes
notes
Smallest model in the post, included because it is the only llama-bench row for Thor in this pass and the command line is reproducible. tg256 was 112.8 tok/s. Stddev on tg128 is wide (+/- 6.4).
Notes
Smallest model in the post, included because it is the only llama-bench row for Thor in this pass and the command line is reproducible. tg256 was 112.8 tok/s. Stddev on tg128 is wide (+/- 6.4).