LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Post-training quantization works down to 8 bits and then breaks down; quantization-aware training needs the original training data, which is unavailable for most released LLMs.

Quantization-aware training made data-free by distilling from the model's OWN generations: the pre-trained model generates the calibration sequence, and those sequences are the student-and-teacher data. Straight-through-estimator gradients on quantized weights, activations and the KV cache simultaneously, with the FP16 frozen copy of the same model as teacher. Being data-free, it works on any released generative model.

Paper's own claim (abstract): LLaMA 7B / 13B / 30B quantized down to 4 bits, with 'large improvements over training-free methods, especially in the low-bit settings'. COMPRESSION RATIO: 4-bit weights + activations + KV cache (16 -> 4 bit on all three). QUALITY COST: not stated as a number in the abstract; the key comparative numbers come from SpinQuant, which reports LLM-QAT trailing by 19.1 points of zero-shot reasoning accuracy at W4A4KV4 on LLaMA-2 7B - i.e. the LLM-QAT 4-bit configuration is roughly 19 points worse than the rotation-based methods, which is a real and often-cited quality gap.

SLUG RENAMED 2026-10-04 (parent agent): was 'llm-qat', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2305.17888 verified via /abs/; single version. Venue ACL 2024 from Semantic Scholar - note this conflicts with the arXiv v1 date of 2023-05-29, so year = venue year 2024. The 19.1-point gap is SpinQuant's reported comparison, recorded here as evidence about THIS paper's quality rather than as its own claim. arXiv ID and title verified by fetching https://arxiv.org/abs/2305.17888 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2305.17888) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'findings' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2024.findings-acl.26, booktitle 'Findings of the Association for Computational Linguistics: ACL 2024'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which.