ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation

No systematic study existed of how PTQ choices (scheme, model family, bit width, weight vs activation) interact across the 125M-176B scale range.

Two parts: (1) a sensitivity study crossing RTN / GPTQ / ZeroQuant and variants over weight-only, activation-only and weight-and-activation quantization on two model families from 125M to 176B; (2) Low-Rank Compensation (LoRC) - after quantizing to the target bit width, fit a rank-r matrix per transformer block to reconstruct the residual quantization error, adding a small amount of parameters to recover quality.

Paper's own NEGATIVE result is the headline: across every method tested, 'none of the current methods can achieve the original model quality for quantization with either INT4-weight or INT4-weight-and-INT8-activation'. LoRC is reported as improving quality recovery with a minimal parameter increase, but the abstract does NOT give a headline compression ratio paired with a paired quality delta, so speedup_reported is deliberately not invented here. Sensitivity finding worth keeping: activation quantization is more susceptible than weight quantization, and smaller models often beat larger ones on activation quantization.

SLUG RENAMED 2026-10-04 (parent agent): was 'zeroquant-v2', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2303.08302 verified via /abs/; v3. Venue: Semantic Scholar returns an empty venue field and the arXiv comment field says only '25 pages, 4 figures' - no venue evidence found, so recorded as arxiv-preprint. Year = arXiv v1 year 2023. This paper is one of the few in the KB whose most useful contribution is a negative result. arXiv ID and title verified by fetching https://arxiv.org/abs/2303.08302 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2303.08302) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.