A100-80GB SXM Llama-3-70B Q4_K_M decode (RunPod measured)
23.0 tok/s, single-GPU llama.cpp GGUF
Fields
id
benchmarks/a100-80gb-sxm-llama-3-70b-q4-k-m-decodeMetric
decode_tok_s
Value
23
Unit
tok/s, single-GPU llama.cpp GGUF
Model
Meta-Llama-3-70B
Methodology
XiongjieDai GPU-Benchmarks-on-LLM-Inference, llama.cpp on RunPod, Q4_K_M GGUF, batch-1 decode. Reported range 22-24 tok/s; midpoint 23 recorded.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
70B Q4_K_M fits entirely in 80GB HBM2. Used as the anchor for estimating the A100-40GB variant's 70B ceiling (~17-19 tok/s) - bandwidth is the binding constraint on dense decode.
Notes
70B Q4_K_M fits entirely in 80GB HBM2. Used as the anchor for estimating the A100-40GB variant's 70B ceiling (~17-19 tok/s) - bandwidth is the binding constraint on dense decode.