A100-80GB SXM Llama-3-8B Q4_K_M decode (RunPod measured)
133.0 tok/s, single-GPU llama.cpp GGUF
Fields
id
benchmarks/a100-80gb-sxm-llama-3-8b-q4-k-m-decodeMetric
decode_tok_s
Value
133
Unit
tok/s, single-GPU llama.cpp GGUF
Model
Meta-Llama-3-8B
Methodology
XiongjieDai GPU-Benchmarks-on-LLM-Inference, llama.cpp on RunPod, Q4_K_M GGUF, batch-1 decode
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Closest measured sibling for the A100 40GB SXM4 record. 80GB part has ~2,039 GB/s vs the 40GB's ~1,555 GB/s (~24% less BW) - scale 8B decode down to ~100-110 tok/s for the 40GB part. Used as the calibration point for the A100-40GB SXM4 row in the used-datacenter ingest.
Notes
Closest measured sibling for the A100 40GB SXM4 record. 80GB part has ~2,039 GB/s vs the 40GB's ~1,555 GB/s (~24% less BW) - scale 8B decode down to ~100-110 tok/s for the 40GB part. Used as the calibration point for the A100-40GB SXM4 row in the used-datacenter ingest.