NVIDIA A100 80GB SXM, llama.cpp (RunPod): Llama-3-8B + 70B Q4_K_M decode tok/s (measured)
133.0 decode tok/s (8B Q4_K_M; 70B Q4_K_M 22-24 t/s)
Fields
id
benchmarks/comm-a100-80gb-sxm-llama3-8b-q4-decodeMetric
decode_tok_s
Value
133
Unit
decode tok/s (8B Q4_K_M; 70B Q4_K_M 22-24 t/s)
Model
Llama-3-8B + Llama-3-70B Q4_K_M (GGUF)
Methodology
THIRD-PARTY MEASURED (XiongjieDai GPU-Benchmarks-on-LLM-Inference, RunPod, llama.cpp). A100 80GB SXM (2039 GB/s): 133 tok/s Llama-3-8B Q4_K_M decode; 22-24 tok/s on 70B Q4_K_M. This is the closest MEASURED sibling to the A100 40GB part - the 40GB has ~24% less bandwidth (1555 vs 2039 GB/s) -> expect ~100-110 t/s 8B and ~17-19 t/s 70B on the 40GB card/module.
Measured by
third_party
Reproducible
yes
notes
The calibration point for the A100 40GB records. A100-80GB-SXM4 is the higher-bandwidth sibling. Used 40GB SXM4 bare module ~$2,499 (compliance-gated, EAR 3A090); used 40GB PCIe card ~$6,995-8,074. Full current CUDA stack (CC8.0). The price gap between SXM4 and PCIe is the HGX baseboard problem priced in.
Notes
The calibration point for the A100 40GB records. A100-80GB-SXM4 is the higher-bandwidth sibling. Used 40GB SXM4 bare module ~$2,499 (compliance-gated, EAR 3A090); used 40GB PCIe card ~$6,995-8,074. Full current CUDA stack (CC8.0). The price gap between SXM4 and PCIe is the HGX baseboard problem priced in.