inference/research
draftlow · 0.50updated 2026-10-15

A100 40GB SXM4 Llama-3-70B Q4_K_M decode (sibling-derived estimate)

18.0 tok/s (ESTIMATE - not measured)
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
benchmarks/comm-a100-40gb-sxm4-est-llama3-70b-q4-decode
Metric
decode_tok_s
Value
18
Unit
tok/s (ESTIMATE - not measured)
Model
Llama-3-70B Q4_K_M (GGUF)
Methodology
DERIVED, not measured. Basis: closest measured sibling A100-80GB SXM does 22-24 tok/s on 70B Q4_K_M (XiongjieDai RunPod llama.cpp measurement); the 40GB part has ~24% less bandwidth (1,555 vs 2,039 GB/s) -> ~17-19 tok/s expected. Recorded at midpoint 18 to make the derivation explicit; treat as a roofline-style estimate, not a run.
Measured by
third_party
notes
Bandwidth-scaled from a measured sibling, per research-used-datacenter.md. Exists so the 'what does a $2,499 SXM4 module do on 70B' question has an honest estimated row rather than a fabricated measured one.

Notes

Bandwidth-scaled from a measured sibling, per research-used-datacenter.md. Exists so the 'what does a $2,499 SXM4 module do on 70B' question has an honest estimated row rather than a fabricated measured one.

Cross-references