llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-70B Q4_K_M pp1024 tok/s (XiongjieDai)

THIRD-PARTY MEASURED (XiongjieDai/GPU-Benchmarks-on-LLM-Inference, RunPod rental, llama.cpp cuBLAS build, ~May 2024). A100 PCIe 80GB (1935 GB/s HBM2e) single card, -ngl 10000 full offload, LLaMA-3 model. tg column = average tok/s generating up to 1024 tokens (batch-1, per-request, NOT aggregate); pp column = average tok/s on a 1024-token prompt eval.

70B prompt eval. The PCIe A100-80GB is the closest MEASURED sibling to the still-uncovered 40GB-PCIe SKU (same GA100, 1555 vs 1935 GB/s — expect ~80% of these figures on the 40GB card).