llama.cpp on 4x NVIDIA A100 PCIe 80GB: LLaMA-3-70B F16 tg1024 tok/s (XiongjieDai)
7.38 tok/s (tg1024, batch-1)
Fields
id
benchmarks/xjd-a100pcie80x4-llama3-70b-f16-tg1024Metric
decode_tok_s
Value
7.38
Unit
tok/s (tg1024, batch-1)
Model
LLaMA-3-70B F16
Methodology
THIRD-PARTY MEASURED (XiongjieDai/GPU-Benchmarks-on-LLM-Inference, RunPod rental, llama.cpp cuBLAS build, ~May 2024). 4x A100 PCIe 80GB x4 parallel, -ngl 10000 full offload, LLaMA-3 model. tg column = average tok/s generating up to 1024 tokens (batch-1, per-request, NOT aggregate); pp column = average tok/s on a 1024-token prompt eval.
Measured by
third_party
Reproducible
yes
notes
70B F16 across 4x80GB: 7.38 t/s — F16 70B is the stress case the affordable multi-A100 rigs actually hit; Q4_K_M on one card is already faster (22.1 t/s) because it stays on-device. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
70B F16 across 4x80GB: 7.38 t/s — F16 70B is the stress case the affordable multi-A100 rigs actually hit; Q4_K_M on one card is already faster (22.1 t/s) because it stays on-device. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).