inference/research
deprecatedlow · 0.20updated 2026-10-04

M5 Max (128GB): Llama 3.3 70B Q4_K_M decode tokens/sec

22.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.
Deprecated. Superseded or no longer accurate.

Fields

id
benchmarks/local-m5-max-70b-q4-decode
Metric
decode_tok_s
Value
22
Unit
tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.
Model
Llama 3.3 70B
Methodology
MLX-LM Metal backend, batch size 1, 512 token prompt, 128 token output, 4-bit quantization, 128GB unified memory
Measured by
third_party
Reproducible
no
notes
M5 Max with 128GB unified memory at 614 GB/s. A 70B model at 4-bit requires ~35GB of weights, fitting comfortably. The 614 GB/s bandwidth yields ~22 tokens/sec for decode. The M5 Max is the highest-performance Apple Silicon laptop chip for local inference. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): format_id holds 'mlx-quantized-4bit', which is not a record id AND has no candidate - data/quantization/ contains no MLX format record. It must NOT be mapped to gguf-q4-k-m even though the benchmark name says Q4_K_M: MLX-LM uses its own 4-bit format and not GGUF, and engines/mlx-lm.json lists mlx-quantized-4bit among its own formats as distinct from GGUF and from AWQ/GPTQ. Left unchanged; the real fix is to add the missing quantization record, which is intake work, not a cross-reference repair. LOAD-CONDITION AUDIT (bq- 2026-10-04) - STATUS: DECOMMISSIONED. THIS IS NOT A MEASUREMENT. WHY THIS RECORD IS POISONOUS RATHER THAN MERELY INCOMPLETE: it looks fully condition-complete. It states batch size 1, a 512-token prompt, a 128-token output and the precision - the exact four things the repo's iron rule demands - and it is still not a benchmark. Those four values are the generic defaults shared verbatim by all 11 records in this family; they describe no run because no run happened. WHAT THE SOURCES ACTUALLY CONTAIN: local-apple-macbook-pro-m5-tech-specs (an Apple tech-specs page) - specification/datasheet documents containing NO decode-throughput measurement of any kind. The recorded figure (22 tok/s) appears nowhere in any cited source. Three of these notes state the reasoning explicitly as bandwidth-derived ('the N GB/s bandwidth yields ~M tokens/sec'), which is an ESTIMATE ARITHMETICALLY PROJECTED from a spec sheet, not a measurement of a model on a part. WHY THE FIGURES ARE ALSO PHYSICALLY IMPLAUSIBLE AS SET: local-rtx-5070-13b-q4-decode records 120 tok/s on a card with 672 GB/s of bandwidth, which is faster than the same family's own 85 tok/s figure for a 1792 GB/s RTX 5090 - a 2.7x bandwidth advantage yielding a LOWER number, and 120 tok/s exceeds what any 8B 4-bit model can pull at that bandwidth. MEASURED_BY: left as 'third_party' ONLY because schemas/benchmark.schema.json forbids null for measured_by and the audit was scoped not to edit schemas/. It is a placeholder, not a fact: nobody third-party measured this. Same for value, which is typed number and cannot be nulled here. Both should become null the moment the schema permits it; until then, status=deprecated plus this note is what stands between this record and a citation. CONFIDENCE 0.9 -> 0.2 and reproducible true -> false: reproducible was unsupportable for a figure with no run behind it. TO RECOVER: an actual llama.cpp or MLX-LM run on the named part, recording the engine build string, model revision, exact prompt/output token counts, whether prompts were VARIED or REPEATED (repeated prompts read 9.5-16.9x high - see ev-llamacpp-spec-bench-replay-inflates), the warmup policy, achieved clocks, and the raw tok/s counter. Until then this record must not be cited as a throughput number. RE-AUDIT 2026-10-04, independent pass (benchmark throughput + latency audit): THE DEPRECATION IS CONFIRMED, on the primary basis rather than on the earlier internal-consistency argument. TWO INDEPENDENT NEGATIVE METHODS, both required before asserting a source lacks a measurement: (1) CORPUS-INTERNAL. Each recorded value was searched for in its own cited source record. 0 occurrences for all 12 citations across the eleven records. The cited records are 7 citations of consumer-nvidia-rtx-blackwell-gpu-architecture-whitepaper (kind: whitepaper) and 5 citations of spec-sheet records (consumer-nvidia-rtx-pro-6000-blackwell-datasheet, consumer-apple-macbook-pro-14-m4-max-tech-specs, local-apple-macbook-pro-m5-tech-specs, local-apple-mac-studio-m5-tech-specs, local-nvidia-rtx-5070-product-page). No benchmark kind is cited anywhere in this family. (2) EXTERNAL, the primary documents themselves, retrieved and keyword-searched: the NVIDIA RTX Blackwell GPU Architecture whitepaper PDF (V1.1) returned 0 occurrences of 'token', 'throughput', 'tok/s', 'decode' and 'inference'; the Apple MacBook Pro M4 Max tech-specs page and the Apple Mac Studio tech-specs page each returned 0 occurrences of 'token', 'throughput', 'tok/s' and 'inference' (their 6 'decode' hits are video-codec rows - 'Video decode engine', 'ProRes encode and decode engine' - not LLM decode). A spec sheet that cannot say 'token' cannot contain a decode-throughput measurement. THE EARLIER 'INVERSION' ARGUMENT WAS WITHDRAWN AS INVALID. It read local-rtx-5070-13b-q4-decode (120 tok/s on 672 GB/s) against local-rtx-5090-32b-q4-decode (85 tok/s on 1792 GB/s) and called the 2.7x bandwidth advantage producing a lower number proof of error. It is not: those two records run DIFFERENT MODELS. The 5070 record loads Llama 3.1 8B at Q4_K_M (~5 GB); the 5090 record loads Qwen3 32B at Q4_K_M (~18 GB). Measured against their own bandwidth roofs they are 89% and 85% respectively - mutually consistent. A slower-card/higher-tok/s comparison is only evidence when the two records hold the model, the quantization and the workload constant, and here nothing does. THE REPLACEMENT TEST IS THE HARD ROOFLINE, which does hold those constant: at batch 1 every decode step must read the model weights at least once, so tok/s cannot exceed bandwidth / weight bytes. Five of these eleven figures exceed their OWN roof - local-rtx-pro-6000-70b-q8-decode 55 tok/s against a 25.6 roof (215%), local-m5-max-70b-q4-decode 22 against 17.5 (125%), local-rtx-5080-32b-q4-decode 65 against 53.3 (122%), local-m4-max-70b-q4-decode 18 against 15.6 (115%), local-m5-ultra-70b-q4-decode 35 against 34.3 (102%). These are dense models, so no MoE correction applies. A Q8_0 70B cannot out-run a Q4_K_M 70B on two parts of identical bandwidth, because Q8_0 moves twice the bytes per token. INSTRUMENT VALIDATED BEFORE USE: the same roofline test was run against six known-good, reproducible third-party and vendor batch-1 decode records in this repo and passes 6 of 6 at 34-86% of roof - after correcting it for mixture-of-experts models, whose divisor is ACTIVE parameters rather than total. Uncorrected, it falsely flags Qwen3-30B-A3B and Qwen3.5-35B-A3B records at 1328% and 336% of roof; anyone reusing this test must apply that correction. THE REMAINING SINGLE-USER/MULTI-USER GAP IS REAL BUT MOOT: these eleven records state batch size 1 and never say per-user vs aggregate. At batch 1 the two coincide, so the omission would not make the number ambiguous - which is precisely the point that makes these records dangerous. The four load conditions are present, mutually consistent, and describe no run. VERDICT: status=deprecated is JUSTIFIED. The value, measured_by and the four load-condition fields cannot be nulled (schema requires a number and forbids null for measured_by), so the record is retained with status=deprecated, the unit prefixed UNSOURCED, DO NOT CITE, confidence 0.2 and reproducible false, plus this note.

Notes

M5 Max with 128GB unified memory at 614 GB/s. A 70B model at 4-bit requires ~35GB of weights, fitting comfortably. The 614 GB/s bandwidth yields ~22 tokens/sec for decode. The M5 Max is the highest-performance Apple Silicon laptop chip for local inference. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): format_id holds 'mlx-quantized-4bit', which is not a record id AND has no candidate - data/quantization/ contains no MLX format record. It must NOT be mapped to gguf-q4-k-m even though the benchmark name says Q4_K_M: MLX-LM uses its own 4-bit format and not GGUF, and engines/mlx-lm.json lists mlx-quantized-4bit among its own formats as distinct from GGUF and from AWQ/GPTQ. Left unchanged; the real fix is to add the missing quantization record, which is intake work, not a cross-reference repair.

LOAD-CONDITION AUDIT (bq- 2026-10-04) - STATUS: DECOMMISSIONED. THIS IS NOT A MEASUREMENT. WHY THIS RECORD IS POISONOUS RATHER THAN MERELY INCOMPLETE: it looks fully condition-complete. It states batch size 1, a 512-token prompt, a 128-token output and the precision - the exact four things the repo's iron rule demands - and it is still not a benchmark. Those four values are the generic defaults shared verbatim by all 11 records in this family; they describe no run because no run happened. WHAT THE SOURCES ACTUALLY CONTAIN: local-apple-macbook-pro-m5-tech-specs (an Apple tech-specs page) - specification/datasheet documents containing NO decode-throughput measurement of any kind. The recorded figure (22 tok/s) appears nowhere in any cited source. Three of these notes state the reasoning explicitly as bandwidth-derived ('the N GB/s bandwidth yields ~M tokens/sec'), which is an ESTIMATE ARITHMETICALLY PROJECTED from a spec sheet, not a measurement of a model on a part. WHY THE FIGURES ARE ALSO PHYSICALLY IMPLAUSIBLE AS SET: local-rtx-5070-13b-q4-decode records 120 tok/s on a card with 672 GB/s of bandwidth, which is faster than the same family's own 85 tok/s figure for a 1792 GB/s RTX 5090 - a 2.7x bandwidth advantage yielding a LOWER number, and 120 tok/s exceeds what any 8B 4-bit model can pull at that bandwidth. MEASURED_BY: left as 'third_party' ONLY because schemas/benchmark.schema.json forbids null for measured_by and the audit was scoped not to edit schemas/. It is a placeholder, not a fact: nobody third-party measured this. Same for value, which is typed number and cannot be nulled here. Both should become null the moment the schema permits it; until then, status=deprecated plus this note is what stands between this record and a citation. CONFIDENCE 0.9 -> 0.2 and reproducible true -> false: reproducible was unsupportable for a figure with no run behind it. TO RECOVER: an actual llama.cpp or MLX-LM run on the named part, recording the engine build string, model revision, exact prompt/output token counts, whether prompts were VARIED or REPEATED (repeated prompts read 9.5-16.9x high - see ev-llamacpp-spec-bench-replay-inflates), the warmup policy, achieved clocks, and the raw tok/s counter. Until then this record must not be cited as a throughput number.

RE-AUDIT 2026-10-04, independent pass (benchmark throughput + latency audit): THE DEPRECATION IS CONFIRMED, on the primary basis rather than on the earlier internal-consistency argument. TWO INDEPENDENT NEGATIVE METHODS, both required before asserting a source lacks a measurement: (1) CORPUS-INTERNAL. Each recorded value was searched for in its own cited source record. 0 occurrences for all 12 citations across the eleven records. The cited records are 7 citations of consumer-nvidia-rtx-blackwell-gpu-architecture-whitepaper (kind: whitepaper) and 5 citations of spec-sheet records (consumer-nvidia-rtx-pro-6000-blackwell-datasheet, consumer-apple-macbook-pro-14-m4-max-tech-specs, local-apple-macbook-pro-m5-tech-specs, local-apple-mac-studio-m5-tech-specs, local-nvidia-rtx-5070-product-page). No benchmark kind is cited anywhere in this family. (2) EXTERNAL, the primary documents themselves, retrieved and keyword-searched: the NVIDIA RTX Blackwell GPU Architecture whitepaper PDF (V1.1) returned 0 occurrences of 'token', 'throughput', 'tok/s', 'decode' and 'inference'; the Apple MacBook Pro M4 Max tech-specs page and the Apple Mac Studio tech-specs page each returned 0 occurrences of 'token', 'throughput', 'tok/s' and 'inference' (their 6 'decode' hits are video-codec rows - 'Video decode engine', 'ProRes encode and decode engine' - not LLM decode). A spec sheet that cannot say 'token' cannot contain a decode-throughput measurement. THE EARLIER 'INVERSION' ARGUMENT WAS WITHDRAWN AS INVALID. It read local-rtx-5070-13b-q4-decode (120 tok/s on 672 GB/s) against local-rtx-5090-32b-q4-decode (85 tok/s on 1792 GB/s) and called the 2.7x bandwidth advantage producing a lower number proof of error. It is not: those two records run DIFFERENT MODELS. The 5070 record loads Llama 3.1 8B at Q4_K_M (~5 GB); the 5090 record loads Qwen3 32B at Q4_K_M (~18 GB). Measured against their own bandwidth roofs they are 89% and 85% respectively - mutually consistent. A slower-card/higher-tok/s comparison is only evidence when the two records hold the model, the quantization and the workload constant, and here nothing does. THE REPLACEMENT TEST IS THE HARD ROOFLINE, which does hold those constant: at batch 1 every decode step must read the model weights at least once, so tok/s cannot exceed bandwidth / weight bytes. Five of these eleven figures exceed their OWN roof - local-rtx-pro-6000-70b-q8-decode 55 tok/s against a 25.6 roof (215%), local-m5-max-70b-q4-decode 22 against 17.5 (125%), local-rtx-5080-32b-q4-decode 65 against 53.3 (122%), local-m4-max-70b-q4-decode 18 against 15.6 (115%), local-m5-ultra-70b-q4-decode 35 against 34.3 (102%). These are dense models, so no MoE correction applies. A Q8_0 70B cannot out-run a Q4_K_M 70B on two parts of identical bandwidth, because Q8_0 moves twice the bytes per token. INSTRUMENT VALIDATED BEFORE USE: the same roofline test was run against six known-good, reproducible third-party and vendor batch-1 decode records in this repo and passes 6 of 6 at 34-86% of roof - after correcting it for mixture-of-experts models, whose divisor is ACTIVE parameters rather than total. Uncorrected, it falsely flags Qwen3-30B-A3B and Qwen3.5-35B-A3B records at 1328% and 336% of roof; anyone reusing this test must apply that correction. THE REMAINING SINGLE-USER/MULTI-USER GAP IS REAL BUT MOOT: these eleven records state batch size 1 and never say per-user vs aggregate. At batch 1 the two coincide, so the omission would not make the number ambiguous - which is precisely the point that makes these records dangerous. The four load conditions are present, mutually consistent, and describe no run. VERDICT: status=deprecated is JUSTIFIED. The value, measured_by and the four load-condition fields cannot be nulled (schema requires a number and forbids null for measured_by), so the record is retained with status=deprecated, the unit prefixed UNSOURCED, DO NOT CITE, confidence 0.2 and reproducible false, plus this note.

Cross-references

Accelerators (1)

Engine (1)

Format (1)