Hardware Corner Jun 2025: Quadro RTX 8000, llama.cpp b5508, Llama 3.3 70B Q5_K_M, decode 7.59 t/s at 8k
7.59 tok/s per-request decode (batch 1, 8192-token context)
Fields
id
benchmarks/llamacpp-b5508-quadro-rtx-8000-llama33-70b-q5km-8k-tgMetric
decode_tok_s
Value
7.59
Unit
tok/s per-request decode (batch 1, 8192-token context)
Model
Llama 3.3 70B Q5_K_M GGUF
Methodology
THIRD-PARTY MEASURED (hardware-corner.net, Allan Witt, updated June 17, 2025). One Quadro RTX 8000, 48 GB GDDR6, NVIDIA driver 570.144, Windows 11, Intel i9-12900K, 128 GB DDR4-3600. llama.cpp build b5508, CUDA 12.8, Q5_K_M GGUF, 8192-token context, full GPU offload. Per request, batch size 1, not an aggregate across concurrent requests. The page prints prompt-processing and token-generation tok/s but not the exact prompt and output lengths inside the 8k context window, and not the repetition count.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Used-market Turing card: 48 GB in one slot at 672 GB/s. The same page says a single RTX 3090 did about 35 t/s decode and 750 t/s prefill on Qwen1.5 32B Q4_K_M at 32k context, against this card 11 t/s and 344 t/s on the Q5_K_M 8k run; those 3090 figures are rounded and at a different quant and context, so they are not filed as their own records.
Notes
Used-market Turing card: 48 GB in one slot at 672 GB/s. The same page says a single RTX 3090 did about 35 t/s decode and 750 t/s prefill on Qwen1.5 32B Q4_K_M at 32k context, against this card 11 t/s and 344 t/s on the Q5_K_M 8k run; those 3090 figures are rounded and at a different quant and context, so they are not filed as their own records.