NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
14.58 tok/s (tg1024 avg, per-request batch-1)
Fields
id
benchmarks/xd-rtx-a6000-70b-q4km-tg1024Metric
decode_tok_s
Value
14.58
Unit
tok/s (tg1024 avg, per-request batch-1)
Model
LLaMA-3 70B Q4_K_M
Methodology
THIRD-PARTY MEASURED (XiongjieDai GPU-Benchmarks-on-LLM-Inference, RunPod rental). llama.cpp built with cuBLAS (make LLAMA_CUBLAS=1), May-2024 snapshot. NVIDIA RTX A6000 48GB full offload (-ngl 10000); llama-bench pp/tg at 512/1024/4096/8192; value is the 1024-token column. Ubuntu 22.04, CUDA 12.1.1. Per-request batch-1, NOT aggregate.
Measured by
third_party
Reproducible
yes
notes
70B Q4_K_M (~40GB weights) fits in the 48 GB framebuffer - this is the single-GPU 70B decode row.
Notes
70B Q4_K_M (~40GB weights) fits in the 48 GB framebuffer - this is the single-GPU 70B decode row.