A100 40GB PCIe Qwen3.5-9B UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
125 tok/s decode (tg128, per-request batch-1); top prefill 4009 tok/s
Fields
id
benchmarks/llamabench-a100pcie40-qwen35-9b-decode-tg128Metric
decode_tok_s
Value
125
Unit
tok/s decode (tg128, per-request batch-1); top prefill 4009 tok/s
Model
Qwen3.5-9B
Methodology
THIRD-PARTY MEASURED, crowdsourced via llamabench.ai (single submitter @wmeddie, n=1 for this model). llama.cpp build b9935, CUDA backend, flash attention ON, weights UD-Q4_K_XL GGUF, f16 KV cache, 0.5k context. Decode figure is the llama-bench tg128 (text-generation 128-token) rate — per-request batch-1, NOT aggregate. Prefill column is the pp prompt-processing tok/s for the same run. No speculative decoding. Single result per model (n=1); no run-to-run variance data.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
First benchmark coverage for nvidia-a100-40gb-pcie (1555 GB/s HBM2e, PCIe Gen4 host link, no NVLink on this card). Decode is bandwidth-bound; MoE parts (Qwen3.6-35B-A3B, gemma-4-26B-A4B, gpt-oss-20b) decode far faster than dense same-size models because only the active expert params stream per token. The 136 tok/s Qwen3.6-35B-A3B figure is the affordable-35B fit-matrix anchor for this card.
Notes
First benchmark coverage for nvidia-a100-40gb-pcie (1555 GB/s HBM2e, PCIe Gen4 host link, no NVLink on this card). Decode is bandwidth-bound; MoE parts (Qwen3.6-35B-A3B, gemma-4-26B-A4B, gpt-oss-20b) decode far faster than dense same-size models because only the active expert params stream per token. The 136 tok/s Qwen3.6-35B-A3B figure is the affordable-35B fit-matrix anchor for this card.