NVIDIA A100, llama.cpp Vulkan (2257758, coopmat2): Llama-2-7B Q4_0 tg128 t/s
160.78 tok/s per-request tg128 (no FA); ±0.16
Fields
id
benchmarks/a100-vulkan-llama27b-q40-nofa-tg128Metric
decode_tok_s
Value
160.78
Unit
tok/s per-request tg128 (no FA); ±0.16
Model
Llama-2-7B Q4_0 GGUF
Methodology
THIRD-PARTY MEASURED (llama.cpp discussion #10879 Vulkan scoreboard, commit 2257758, coopmat2). Llama-2-7B Q4_0, -ngl 99 -fa 0, tg128 = 160.78 ± 0.16 t/s. Scoreboard lists the chip only as 'Nvidia A100' — filed under the 40GB SXM4 record; the 40GB PCIe and 80GB SKUs share the same GA100 die and differ mainly in HBM capacity, so the decode figure is representative to ~first order.
Measured by
third_party
Reproducible
yes
notes
VARIANT AMBIGUITY: scoreboard does not record which A100 SKU (40GB PCIe / 40GB SXM4 / 80GB). Decode tok/s is bandwidth-bound; all A100 SKUs are 1555-2039 GB/s so the figure is broadly representative but the 80GB HBM2e SKU would edge higher.
Notes
VARIANT AMBIGUITY: scoreboard does not record which A100 SKU (40GB PCIe / 40GB SXM4 / 80GB). Decode tok/s is bandwidth-bound; all A100 SKUs are 1555-2039 GB/s so the figure is broadly representative but the 80GB HBM2e SKU would edge higher.