llama.cpp CUDA on NVIDIA A10G (A10 AWS variant): Llama-2-7B Q4_0 FA-on tg128 tok/s
111.51 tok/s (tg128, per-request, batch 1)
Fields
id
benchmarks/llamacpp-cuda-a10g-llama27b-q40-fa-tg128Metric
decode_tok_s
Value
111.51
Unit
tok/s (tg128, per-request, batch 1)
Model
llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params)
Methodology
llama.cpp discussion #15013 comment by @Hastwell (2026-07-19): A10G custom SKU (sm_86, 24GB GDDR6, 498 GB/s), build 571d0d5, -ngl 99 -fa 1. tg128 111.51, pp512 2190.43. A10G is the AWS-sourced 300W 2-slot A10 derivative — same silicon and memory system as the A10, so the number applies to the A10 family with the caveat that thermals differ.
Measured by
third_party
Reproducible
yes
notes
NOT the scoreboard-maintained row (comment data). A10G SKU is a custom variant — kept draft pending a plain-A10 run.
Notes
NOT the scoreboard-maintained row (comment data). A10G SKU is a custom variant — kept draft pending a plain-A10 run.