A100 40GB SXM4 Llama-3-70B Q4_K_M decode (sibling-derived estimate)
18.0 tok/s (ESTIMATE - not measured)
Fields
id
benchmarks/comm-a100-40gb-sxm4-est-llama3-70b-q4-decodeMetric
decode_tok_s
Value
18
Unit
tok/s (ESTIMATE - not measured)
Model
Llama-3-70B Q4_K_M (GGUF)
Methodology
DERIVED, not measured. Basis: closest measured sibling A100-80GB SXM does 22-24 tok/s on 70B Q4_K_M (XiongjieDai RunPod llama.cpp measurement); the 40GB part has ~24% less bandwidth (1,555 vs 2,039 GB/s) -> ~17-19 tok/s expected. Recorded at midpoint 18 to make the derivation explicit; treat as a roofline-style estimate, not a run.
Measured by
third_party
notes
Bandwidth-scaled from a measured sibling, per research-used-datacenter.md. Exists so the 'what does a $2,499 SXM4 module do on 70B' question has an honest estimated row rather than a fabricated measured one.
Notes
Bandwidth-scaled from a measured sibling, per research-used-datacenter.md. Exists so the 'what does a $2,499 SXM4 module do on 70B' question has an honest estimated row rather than a fabricated measured one.