llama.cpp Metal on apple-m1-ultra (48 GPU cores): Llama-2-7B Q8_0 tg128 (scoreboard #4167, 8e672ef)
55.69 tok/s (decode, per-request, batch 1, tg128)
Fields
id
benchmarks/llamacpp-metal-m1-ultra-48core-llama27b-q8-0-tg128-8e672efMetric
decode_tok_s
Value
55.69
Unit
tok/s (decode, per-request, batch 1, tg128)
Model
Llama-2-7B
Methodology
COMMUNITY-SCOREBOARD (llama.cpp discussion #4167, ggerganov-maintained summary table). llama-bench -hf ggml-org/Llama-2-7B-GGUF -p 512 -n 128 -ngl 99 --delay 30, Metal backend, full GPU offload. Build commit 8e672ef (2023-11-21 baseline, before the later Metal flash-attention speedups the same thread documents on M2 Ultra). pp512 = prefill at batch 512; tg128 = batch-1 decode of 128 tokens. Mean over the default repeats. Per-request, NOT aggregate. This row is the LOWER GPU-core bin of the same chip; the higher-core bin is already stored and must not be averaged with this one.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
Lower GPU-core bin from the #4167 summary table. The higher-core bin of this same chip is a separate already-stored record; do not treat these as duplicates. 8e672ef predates the Metal FA improvements, so a current build is faster — the thread's own M2 Ultra re-run shows that.
Notes
Lower GPU-core bin from the #4167 summary table. The higher-core bin of this same chip is a separate already-stored record; do not treat these as duplicates. 8e672ef predates the Metal FA improvements, so a current build is faster — the thread's own M2 Ultra re-run shows that.