llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-70B F16 tg1024 tok/s (XiongjieDai)
7.38 tok/s (tg1024, batch-1 per-request)
Fields
id
benchmarks/xjd-a100pcie80-llama3-70b-f16-tg1024Metric
decode_tok_s
Value
7.38
Unit
tok/s (tg1024, batch-1 per-request)
Model
LLaMA-3-70B F16
Methodology
THIRD-PARTY MEASURED (XiongjieDai/GPU-Benchmarks-on-LLM-Inference, RunPod rental, llama.cpp cuBLAS build, ~May 2024). A100 PCIe 80GB (1935 GB/s HBM2e) single card, -ngl 10000 full offload, LLaMA-3 model. tg column = average tok/s generating up to 1024 tokens (batch-1, per-request, NOT aggregate); pp column = average tok/s on a 1024-token prompt eval.
Measured by
third_party
Reproducible
yes
notes
70B F16 barely fits 80 GB — 7.4 t/s. The PCIe A100-80GB is the closest MEASURED sibling to the still-uncovered 40GB-PCIe SKU (same GA100, 1555 vs 1935 GB/s — expect ~80% of these figures on the 40GB card).
Notes
70B F16 barely fits 80 GB — 7.4 t/s. The PCIe A100-80GB is the closest MEASURED sibling to the still-uncovered 40GB-PCIe SKU (same GA100, 1555 vs 1935 GB/s — expect ~80% of these figures on the 40GB card).