M3 Max Llama-2-7B Q4_0 tg128 (llama.cpp #4167)
66.31 tok/s (tg128, batch 1, per-request)
Fields
id
benchmarks/llamacpp-metal-m3-max-llama27b-q4-0-tg128-8e672efMetric
decode_tok_s
Value
66.31
Unit
tok/s (tg128, batch 1, per-request)
Model
Llama-2-7B
Methodology
llama.cpp discussion #4167 (ggerganov table, commit 8e672ef / ymcui 55978ce). M3 Max 40-core GPU, Metal backend, ngl=99, llama-bench. Llama-2-7B. tg128 = batch-1 decode tok/s; pp512 = batch-512 prefill tok/s.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Apple Silicon M3 Max 40 GPU cores, 400 GB/s unified memory.
Notes
Apple Silicon M3 Max 40 GPU cores, 400 GB/s unified memory.