llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-8B F16 tg1024 tok/s (XiongjieDai)
54.56 tok/s (tg1024, batch-1 per-request)
Fields
id
benchmarks/xjd-a100pcie80-llama3-8b-f16-tg1024Metric
decode_tok_s
Value
54.56
Unit
tok/s (tg1024, batch-1 per-request)
Model
LLaMA-3-8B F16
Methodology
THIRD-PARTY MEASURED (XiongjieDai/GPU-Benchmarks-on-LLM-Inference, RunPod rental, llama.cpp cuBLAS build, ~May 2024). A100 PCIe 80GB (1935 GB/s HBM2e) single card, -ngl 10000 full offload, LLaMA-3 model. tg column = average tok/s generating up to 1024 tokens (batch-1, per-request, NOT aggregate); pp column = average tok/s on a 1024-token prompt eval.
Measured by
third_party
Reproducible
yes
notes
F16 weights (~16 GB) halve decode vs Q4_K_M. The PCIe A100-80GB is the closest MEASURED sibling to the still-uncovered 40GB-PCIe SKU (same GA100, 1555 vs 1935 GB/s — expect ~80% of these figures on the 40GB card).
Notes
F16 weights (~16 GB) halve decode vs Q4_K_M. The PCIe A100-80GB is the closest MEASURED sibling to the still-uncovered 40GB-PCIe SKU (same GA100, 1555 vs 1935 GB/s — expect ~80% of these figures on the 40GB card).