llama.cpp+ZenDNN on EPYC 9755 128c: Llama-3.1-8B-Instruct None end-to-end tok/s (512in/128out/128seq)
914.85 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)
Fields
id
benchmarks/amd-epyc9755-zendnn-llama31-8b-bf16-e2eMetric
tps_aggregate
Value
914.85
Unit
tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)
Model
Llama-3.1-8B-Instruct
Methodology
VENDOR MEASURED (AMD developer blog, ZD-068, Aug 18 2026). AMD EPYC 9755 128-core single socket (one NUMA node pinned, SMT off, 4.06GHz boost, DDR5-6400 12-ch); llama.cpp build b9326, ZenDNN 5.2.2 backend (-DGGML_ZENDNN=ON), ZENDNNL_MATMUL_ALGO=AOCL-DLP, llama-batched-bench 512in/128out/128 parallel seq, flash-attn on, KV cache = model precision, 3-run mean. Llama-3.1-8B-Instruct (BF16): ZenDNN 914.85 vs native-CPU 632.17 tok/s total end-to-end. AGGREGATE over 128 seq.
Measured by
vendor
Reproducible
yes
notes
Native 632.17 -> ZenDNN 914.85 tok/s (1.45x) total throughput. The EPYC-CPU inference path: usable for 70B-class at Q8_0/BF16 batch jobs; see comm-epyc-9554-* for the channel-bound decode case.
Notes
Native 632.17 -> ZenDNN 914.85 tok/s (1.45x) total throughput. The EPYC-CPU inference path: usable for 70B-class at Q8_0/BF16 batch jobs; see comm-epyc-9554-* for the channel-bound decode case.