llama.cpp on 4x NVIDIA A100 PCIe 80GB: LLaMA-3-8B Q4_K_M pp1024 tok/s (XiongjieDai)
8889.35 tok/s (pp1024, 4-GPU parallel split)
Fields
id
benchmarks/xjd-a100pcie80x4-llama3-8b-q4km-pp1024Metric
prefill_tok_s
Value
8,889.35
Unit
tok/s (pp1024, 4-GPU parallel split)
Model
LLaMA-3-8B Q4_K_M GGUF
Methodology
THIRD-PARTY MEASURED (XiongjieDai/GPU-Benchmarks-on-LLM-Inference, RunPod rental, llama.cpp cuBLAS build, ~May 2024). 4x A100 PCIe 80GB x4 parallel, -ngl 10000 full offload, LLaMA-3 model. tg column = average tok/s generating up to 1024 tokens (batch-1, per-request, NOT aggregate); pp column = average tok/s on a 1024-token prompt eval.
Measured by
third_party
Reproducible
yes
notes
4-way layer split: prefill scales to 8889 t/s vs 5800 single — multi-GPU buys prompt throughput, not decode (tg 117.3 vs 138.3 single, decode goes DOWN across split as expected for a BW-bound layer-shuffled model).
Notes
4-way layer split: prefill scales to 8889 t/s vs 5800 single — multi-GPU buys prompt throughput, not decode (tg 117.3 vs 138.3 single, decode goes DOWN across split as expected for a BW-bound layer-shuffled model).