Geerling llama.cpp CUDA on rtx2080ti (pc host): llama-2-13b Q4_K_M tg128 60.51 tok/s
60.51 tok/s (tg128)
Fields
id
benchmarks/geerling-cuda-rtx2080ti-pc-llama-2-13b-q4km-tg128Metric
decode_tok_s
Value
60.51
Unit
tok/s (tg128)
Model
llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params)
Methodology
THIRD-PARTY MEASURED (Jeff Geerling, geerlingguy/ai-benchmarks). llama.cpp build f9ec8858e (7474), backend CUDA, -fa 1 -mmp 0 -ngl 99 (flash attention on, mmap off). Host: Intel Core Ultra 265K, 64GB DDR5, Ubuntu 25.10. Model file size 7.33 GiB, 13.02 B params, Q4_K_M. tg128 mean decode tok/s, per-request, batch 1, generation length 128. NOT aggregate. Whole-system wall power during the run 325.3 W (node scope, not device).
Measured by
third_party
Reproducible
yes
Mean power (W)
325.3
Power scope
node
Thermal state
unknown
notes
Source issue prints the llama-bench mean and the +/- spread; the stored value is the mean. The Pi-host rows are the same physical GPU on a PCIe-constrained host, so they are not a card ceiling.
Notes
Source issue prints the llama-bench mean and the +/- spread; the stored value is the mean. The Pi-host rows are the same physical GPU on a PCIe-constrained host, so they are not a card ceiling.