llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
83.06 tok/s (tg128, batch 1, per-request)
Fields
id
benchmarks/llamacpp-metal-m4-max-40core-llama27b-q8-0-tg128-8e672efMetric
decode_tok_s
Value
83.06
Unit
tok/s (tg128, batch 1, per-request)
Model
Llama-2-7B gguf-q8-0.gguf
Methodology
COMMUNITY-SCOREBOARD (llama.cpp discussion #4167, ggerganov-maintained). llama-bench -hf ggml-org/Llama-2-7B-GGUF -p 512 -n 128 -ngl 99 --delay 30, Metal backend, full GPU offload. Build commit 8e672ef. pp512 = prefill at batch 512; tg128 = batch-1 decode. mean over default -r 5. Per-request, NOT aggregate. Chip: 40 GPU cores, 546 GB/s memory BW. This is the core-count bin the earlier ingest of this table did not file.
Measured by
third_party
Reproducible
yes
notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167). Same Llama-2-7B weights and pp512/tg128 protocol as the other rows of that table. Lower-core bin of the same chip family, so it is not interchangeable with the higher-core row already stored.
Notes
Maintainer-curated Apple Silicon llama-bench table (discussion #4167). Same Llama-2-7B weights and pp512/tg128 protocol as the other rows of that table. Lower-core bin of the same chip family, so it is not interchangeable with the higher-core row already stored.