A100 40GB PCIe llama.cpp b9935: Qwen3.5-4B UD-Q4_K_XL decode
166 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 5306 t/s
Fields
id
benchmarks/llamabench-a100pcie40-qwen35-4b-q4kxl-tg128Metric
decode_tok_s
Value
166
Unit
tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 5306 t/s
Model
Qwen3.5-4B UD-Q4_K_XL
Methodology
llamabench.ai community row, reporter @wmeddie: llama.cpp build b9935, flash-attention on, KV cache f16, 0.5k context depth, no speculative decoding. Decode = tg128 single-stream (per-request, batch-1); prefill = site-reported figure. n=1 per model row — crowd-sourced, not a curated scoreboard.
Measured by
third_party
notes
First measured coverage for the 40GB PCIe SKU. Compare the 80GB PCIe sibling (xiongjiedai RunPod, llama-3-8b Q4_K_M 138.3 t/s): the 40GB card's identical 1555 GB/s bandwidth puts its decode in the same class — consistent with the 125-187 t/s seen here on 7-20B models.
Notes
First measured coverage for the 40GB PCIe SKU. Compare the 80GB PCIe sibling (xiongjiedai RunPod, llama-3-8b Q4_K_M 138.3 t/s): the 40GB card's identical 1555 GB/s bandwidth puts its decode in the same class — consistent with the 125-187 t/s seen here on 7-20B models.