llama.cpp+ZenDNN on EPYC 9755 128c: Mistral-Small-3.2-24B-Instruct-2506 gguf-q8-0 prefill tok/s (512in/128seq)
620.85 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)
Fields
id
benchmarks/amd-epyc9755-zendnn-mistralsmall-24b-q8-pp512Metric
prefill_tok_s
Value
620.85
Unit
tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)
Model
Mistral-Small-3.2-24B-Instruct-2506
Methodology
VENDOR MEASURED (AMD developer blog, ZD-066/067, Aug 18 2026). AMD EPYC 9755 128-core single socket (one NUMA node pinned, SMT off, 4.06GHz boost, DDR5-6400 12-ch); llama.cpp build b9326, ZenDNN 5.2.2 backend (-DGGML_ZENDNN=ON), ZENDNNL_MATMUL_ALGO=AOCL-DLP, llama-batched-bench 512in/128out/128 parallel seq, flash-attn on, KV cache = model precision, 3-run mean. Mistral-Small-3.2-24B-Instruct-2506 (gguf-q8-0): ZenDNN 620.85 vs native-CPU 237.18 tok/s prefill. AGGREGATE over 128 seq, not per-request.
Measured by
vendor
Reproducible
yes
notes
Native-CPU baseline 237.18 tok/s -> ZenDNN 620.85 tok/s (2.62x). Vendor-published, single-socket pinned. Decode falls back to CPU path on ZenDNN 5.2.2 (small-batch not yet accelerated).
Notes
Native-CPU baseline 237.18 tok/s -> ZenDNN 620.85 tok/s (2.62x). Vendor-published, single-socket pinned. Decode falls back to CPU path on ZenDNN 5.2.2 (small-batch not yet accelerated).