llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-f16 pp512 tok/s (scoreboard #4167, 60081bb)
1310.78 tok/s (pp512, prefill batch 512, per-request)
Fields
id
benchmarks/llamacpp-metal-m5-pro-16core-llama27b-f16-pp512-60081bbMetric
prefill_tok_s
Value
1,310.78
Unit
tok/s (pp512, prefill batch 512, per-request)
Model
Llama-2-7B gguf-f16.gguf
Methodology
COMMUNITY-SCOREBOARD (llama.cpp discussion #4167, ggerganov-maintained). llama-bench -hf ggml-org/Llama-2-7B-GGUF -p 512 -n 128 -ngl 99 --delay 30, Metal backend, full GPU offload. Build commit 60081bb. pp512 = prefill at batch 512; tg128 = batch-1 decode. mean over default -r 5. Per-request, NOT aggregate. Chip: 16 GPU cores, 307 GB/s memory BW. This is the core-count bin the earlier ingest of this table did not file.
Measured by
third_party
Reproducible
yes
notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167). Same Llama-2-7B weights and pp512/tg128 protocol as the other rows of that table. Lower-core bin of the same chip family, so it is not interchangeable with the higher-core row already stored.
Notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167). Same Llama-2-7B weights and pp512/tg128 protocol as the other rows of that table. Lower-core bin of the same chip family, so it is not interchangeable with the higher-core row already stored.