SqueezeLLM: Dense-and-Sparse Quantization

Single-batch generative LLM inference is memory-bandwidth-bound, so weights must shrink - but uniform low-bit quantization loses perplexity badly at 3 bits.

Two components: (1) sensitivity-based NON-UNIFORM quantization - per-row bit precision allocation searched from second-order (Fisher-style) sensitivity information so sensitive rows get more bits; (2) the Dense-and-Sparse decomposition - outlier and sensitive weight values are stored in a sparse residual format while the bulk is stored as dense low-bit codebooks, cutting both bytes and dequantization work.

Paper's own claim (abstract, LLaMA family): lossless (to their definition) compression at up to 3 bits, and at 3-bit the perplexity gap from the FP16 baseline is reduced 'by up to 2.1x' versus state-of-the-art methods at the same memory budget. COMPRESSION RATIO: up to ~5.3x at 3 bits vs FP16. QUALITY COST: 'lossless' is the paper's own wording at their threshold; the concrete number to trust is the 2.1x reduction in the perplexity gap. The paper's framing that inference is bandwidth- not compute-bound for batch size 1 is the load-bearing insight and it generalizes beyond this method.

SLUG RENAMED 2026-10-04 (parent agent): was 'squeezellm', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs (['scheme']) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2306.07629 verified via /abs/; v4. Venue ICML 2024 confirmed by the arXiv comment field and Semantic Scholar. Year = venue year 2024 (v1 2023-06-13). The repo already has source qlab-squeezellm-paper for this paper. Sparse+codebook format needs a bespoke kernel, hence research-only. arXiv ID and title verified by fetching https://arxiv.org/abs/2306.07629 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.07629) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.