Hardware Corner Jun 2025: Quadro RTX 8000, ExLlamaV2 0.2.9, Qwen1.5 32B 5.0 bpw, decode 14.2 t/s at 8k
14.2 tok/s per-request decode (8192-token context, 8-bit cache)
Fields
id
benchmarks/exllamav2-quadro-rtx-8000-qwen15-32b-5bpw-8k-tgMetric
decode_tok_s
Value
14.2
Unit
tok/s per-request decode (8192-token context, 8-bit cache)
Model
Qwen1.5 32B EXL2 5.0 bpw
Methodology
THIRD-PARTY MEASURED (hardware-corner.net, Allan Witt, updated June 17, 2025). One Quadro RTX 8000, 48 GB GDDR6, NVIDIA driver 570.144, Windows 11, Intel i9-12900K, 128 GB DDR4-3600. ExLlamaV2 0.2.9, 5.0 bpw EXL2, 8192-token context, max_batch_size 1024, cache_8bit True, one GPU, split none. Per request, not an aggregate across concurrent requests. The page prints prompt-processing and token-generation tok/s but not the exact prompt and output lengths inside the 8k context window, and not the repetition count.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Used-market Turing card: 48 GB in one slot at 672 GB/s. The same page says a single RTX 3090 did about 35 t/s decode and 750 t/s prefill on Qwen1.5 32B Q4_K_M at 32k context, against this card 11 t/s and 344 t/s on the Q5_K_M 8k run; those 3090 figures are rounded and at a different quant and context, so they are not filed as their own records.
Notes
Used-market Turing card: 48 GB in one slot at 672 GB/s. The same page says a single RTX 3090 did about 35 t/s decode and 750 t/s prefill on Qwen1.5 32B Q4_K_M at 32k context, against this card 11 t/s and 344 t/s on the Q5_K_M 8k run; those 3090 figures are rounded and at a different quant and context, so they are not filed as their own records.