inference/research
contestedlow · 0.60updated 2026-10-03

GGUF Q5_0 GSM8K 'delta' vs f16 - quantization scoring ABOVE baseline

1.45 accuracy_delta_pp (GSM8K percentage points, quantized MINUS f16 - i.e. scored ABOVE baseline)

Fields

id
benchmarks/gguf-q5-0-gsm8k-accuracy-delta-llama3-1-8b
Metric
quality
Value
1.45
Unit
accuracy_delta_pp (GSM8K percentage points, quantized MINUS f16 - i.e. scored ABOVE baseline)
Model
Llama-3.1-8B-Instruct (Q5_0 GGUF)
Methodology
Unified llama.cpp GGUF study, Table 2, GSM8K column. Model Llama-3.1-8B-Instruct as Q5_0 GGUF, 65.19 percent size reduction. Metric: GSM8K flexible-extract via lm-evaluation-harness v0.4.9.2, 5-shot, default settings, against the same checkpoint's F16 GGUF baseline. F16 scores 77.63; Q5_0 scores 79.08.
Measured by
third_party
Reproducible
yes
notes
DELTA = +1.45 pp - QUANTIZATION SCORED ABOVE THE BASELINE. Deliberately included as a trap record: a positive quality delta is not an improvement, it is measurement noise. The study flags it itself - 'these small gains over F16 should be interpreted cautiously', noting that finite benchmark sets introduce variance even under deterministic decoding and that modest improvements can reflect idiosyncrasies of the scoring pipeline rather than true superiority. Corroborating evidence in the same column: Q5_1 +0.84, Q5_K_M +0.91, Q6_K +0.54, Q4_K_S -0.30, Q3_K_L -3.56. The 5-bit formats scatter on both sides of zero with no monotone structure, which is the signature of noise rather than signal. Do not use this record to argue Q5_0 improves reasoning. Q5_0 IS a defensible deployment choice on the study's own Pareto analysis (65.19 percent reduction at roughly -0.65 percent AvgLoss, the best quality-per-reduction point), but the honest quality statement is 'indistinguishable from f16 within benchmark noise'. Second caveat: Q5_0 is a LEGACY symmetric block-5 format (K=32, scale only, no zero point), not a K-quant; format_id points at gguf-q5-k-m as the nearest existing record and the mismatch is recorded rather than hidden.

Notes

DELTA = +1.45 pp - QUANTIZATION SCORED ABOVE THE BASELINE. Deliberately included as a trap record: a positive quality delta is not an improvement, it is measurement noise. The study flags it itself - 'these small gains over F16 should be interpreted cautiously', noting that finite benchmark sets introduce variance even under deterministic decoding and that modest improvements can reflect idiosyncrasies of the scoring pipeline rather than true superiority. Corroborating evidence in the same column: Q5_1 +0.84, Q5_K_M +0.91, Q6_K +0.54, Q4_K_S -0.30, Q3_K_L -3.56. The 5-bit formats scatter on both sides of zero with no monotone structure, which is the signature of noise rather than signal. Do not use this record to argue Q5_0 improves reasoning. Q5_0 IS a defensible deployment choice on the study's own Pareto analysis (65.19 percent reduction at roughly -0.65 percent AvgLoss, the best quality-per-reduction point), but the honest quality statement is 'indistinguishable from f16 within benchmark noise'. Second caveat: Q5_0 is a LEGACY symmetric block-5 format (K=32, scale only, no zero point), not a K-quant; format_id points at gguf-q5-k-m as the nearest existing record and the mismatch is recorded rather than hidden.

Cross-references

Engine (1)

Format (1)