llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
270.37 tok/s (pp512, batch 512, per-request)
Fields
id
benchmarks/llamacpp-metal-m1-pro-16core-llama27b-q8-0-pp512-8e672efMetric
prefill_tok_s
Value
270.37
Unit
tok/s (pp512, batch 512, per-request)
Model
Llama-2-7B gguf-q8-0.gguf
Methodology
COMMUNITY-SCOREBOARD (llama.cpp discussion #4167, ggerganov-maintained). llama-bench -hf ggml-org/Llama-2-7B-GGUF -p 512 -n 128 -ngl 99 --delay 30, Metal backend, full GPU offload. Build commit 8e672ef. pp512 = prefill of one prompt at batch 512; tg128 = batch-1 decode of 128 tokens. mean over the default -r 5. Per-request, NOT aggregate. Chip: Apple M1 Pro (14- or 16-core GPU), 16 GPU cores, 200 GB/s unified-memory bandwidth as printed in the scoreboard.
Measured by
third_party
Reproducible
yes
notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167), the same Llama-2-7B weights and pp512/tg128 protocol as the CUDA/ROCm scoreboards. This row is the 16-core GPU bin; the other bin of the same die is a separate record. F16 cells the scoreboard leaves blank are not filed.
Notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167), the same Llama-2-7B weights and pp512/tg128 protocol as the CUDA/ROCm scoreboards. This row is the 16-core GPU bin; the other bin of the same die is a separate record. F16 cells the scoreboard leaves blank are not filed.