EPYC 9554 Llama-3.1-8B-Instruct Q4_K_M tg128 (ahelpme)
49.64 tok/s (tg decode, batch 1, per-request)
Fields
id
benchmarks/ah-epyc9554-llama31-8b-q4km-tg128Metric
decode_tok_s
Value
49.64
Unit
tok/s (tg decode, batch 1, per-request)
Model
Llama-3.1-8B-Instruct
Methodology
ahelpme.com measured llama-bench, AMD EPYC 9554 Genoa 64C single-socket, 12ch DDR5-5600 (~460.8 GB/s), --numa distribute, -t 64, -p 0, -n 128..2048. BLAS+RPC backend, tg128 decode tok/s batch-1, llama.cpp build 51f311e0 (b4753).
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Single-socket Genoa, all 12 DDR5 channels populated. CPU-only decode; memory-bandwidth-bound.
Notes
Single-socket Genoa, all 12 DDR5 channels populated. CPU-only decode; memory-bandwidth-bound.