Fully-populated EPYC Genoa 12ch on a 405B Q4 model (derived feasibility number)
1.75 tok/s (ESTIMATE - derived)
Fields
id
benchmarks/comm-epyc-genoa-405b-q4-decode-estMetric
decode_tok_s
Value
1.75
Unit
tok/s (ESTIMATE - derived)
Model
Llama-3.1-405B-class Q4 (~230GB weights)
Methodology
DERIVED, not measured. ~460 GB/s (12ch DDR5-4800 fully populated) / ~230 GB weights-per-token read -> ~1.5-2 tok/s expected on a 768GB Genoa/Turin box; recorded at midpoint 1.75. ik_llama.cpp fused MoE/AVX-512 paths worth ~1.9x on MoE decode over mainline.
Measured by
third_party
notes
The one legit CPU-inference niche: a used Milan/Genoa box + ik_llama.cpp is the cheapest way to RUN a 400B-class model at all - batch/overnight-only speed, but it fits, which no consumer GPU can claim. RAM is the cost wall in 2026 (64GB DDR5 RDIMMs ~$873 -> projected ~$1,586).
Notes
The one legit CPU-inference niche: a used Milan/Genoa box + ik_llama.cpp is the cheapest way to RUN a 400B-class model at all - batch/overnight-only speed, but it fits, which no consumer GPU can claim. RAM is the cost wall in 2026 (64GB DDR5 RDIMMs ~$873 -> projected ~$1,586).