inference/research
draftlow · 0.40minorupdated 2026-10-04

Community consensus on where the quantization quality cliff is: below Q4 is unusable, and below Q8 is unusable for code - but this is forum opinion, and the one measurement in the thread does not test it

[minor] format
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
gotchas/comm-community-quant-quality-floor-opinion
Class
format
Severity
minor
Concepts
ggufq4q5q6q8quantization-qualitycoding-tasks
Symptom
The recurring practitioner judgement in r/LocalLLaMA, stated plainly by timschwartz: 'Quantization is lossy compression. It shaves off a few IQ points in exchange for making it smaller. Stuff below Q4 is usually too smooth-brained to be useful.' The sharper version, from Randommaggy: 'For certain things even mild quantization makes the models go from occasionally useful to absolutely useless. For coding anything below Q8 is beyond useless unless it can plagiarize an answer wholesale.' Both are given as answers to a newcomer's question about whether Q4 vs Q8 matters.
Root cause
No root cause is established by these reports, and this record does not invent one. The stated mechanism is generic lossy compression, and the domain-dependence claim (code is hit harder than prose) is plausible and widely repeated but is not supported by anything in this thread. The general shape is consistent with the perplexity-delta records elsewhere in this database, which measure small quality losses at Q4_K_M; what the community adds is the observation that aggregate perplexity understates damage on narrow, format-constrained tasks.
Workaround
Treat Q4 as the practical floor for general chat and prose, and test at or above Q6 if the workload is code or anything requiring strict instruction following. This is a starting hypothesis to verify against the quality-delta records, not a substitute for measuring your own workload. Do not size hardware off this thread - none of these commenters gives a model, a task suite, or a baseline.
notes
REPORTER COUNT: TWO distinct opinions (timschwartz, Randommaggy), plus Ars-Paradox giving the same direction as an analogy rather than an experience ('Q8 is like 1440p and Q4 is like 720p'). The OP (mrmojoer) ran the thread's only measurement and it does NOT support the claims being discussed: on a 3060 12GB they compared llama2-7b-chat-q6_k, llama2-13b-chat-q6_k, mistral-7b-instruct-q6_k and mistral-7b-instruct-q8_0 on one task (list 53 'wh'-words, numbered and alphabetical). Llama2 gave 25 words and lost both constraints after the first few; Mistral gave 53 but lost them after ~15-17; ChatGPT-4 fulfilled all requirements first try. All four local models were Q6_K or Q8_0 - no Q4 anywhere - and it is n=1 per model on a single task with no perplexity or eval-suite measurement. So the measurement is recorded here only to be set aside: it is not evidence about Q4 vs Q8, and it is not evidence that Mistral beats Llama2 either. The honest summary is that this thread is strong evidence of what practitioners BELIEVE about the quality cliff and weak-to-no evidence of where it actually is. Status is draft for that reason.

Notes

REPORTER COUNT: TWO distinct opinions (timschwartz, Randommaggy), plus Ars-Paradox giving the same direction as an analogy rather than an experience ('Q8 is like 1440p and Q4 is like 720p'). The OP (mrmojoer) ran the thread's only measurement and it does NOT support the claims being discussed: on a 3060 12GB they compared llama2-7b-chat-q6_k, llama2-13b-chat-q6_k, mistral-7b-instruct-q6_k and mistral-7b-instruct-q8_0 on one task (list 53 'wh'-words, numbered and alphabetical). Llama2 gave 25 words and lost both constraints after the first few; Mistral gave 53 but lost them after ~15-17; ChatGPT-4 fulfilled all requirements first try. All four local models were Q6_K or Q8_0 - no Q4 anywhere - and it is n=1 per model on a single task with no perplexity or eval-suite measurement. So the measurement is recorded here only to be set aside: it is not evidence about Q4 vs Q8, and it is not evidence that Mistral beats Llama2 either. The honest summary is that this thread is strong evidence of what practitioners BELIEVE about the quality cliff and weak-to-no evidence of where it actually is. Status is draft for that reason.

Cross-references

Affects (2)