CPU/EPYC inference collapses unless every memory channel is populated
[major] config
Fields
id
gotchas/gotcha-cpu-inference-must-populate-all-memory-channelsClass
config
Severity
major
Concepts
memory-channelsnumaepycbandwidth
Symptom
A Genoa box with ONE 64GB DIMM measured 1.5 tok/s on 34B Q4_0, jumping to 4 tok/s with a second stick - one DIMM wastes ~92% of the bandwidth you paid for.
Root cause
LLM decode is bandwidth-bound; server CPUs only reach their rated ~460 GB/s (Genoa 12ch) with a DIMM on every channel. A few fat DIMMs leave most channels idle.
Workaround
Budget 12x 32GB or 12x 64GB, not 4x 96GB. Fully populated Genoa lands ~7 tok/s on 70B Q4_K_M. Dual-socket does NOT double it - NUMA splits to ~1.3-1.6x single-socket.