SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression

3-4 bit quantization causes moderate-to-high accuracy loss, worst on the small (1-10B) models that edge deployment actually targets.

Store the weights that cause the largest quantization errors (outlier weights) SEPARATELY in a sparse high-precision structure, and compress everything else to 3-4 bits. A sensitivity criterion identifies outlier weights per group; the sparse buffer is stored as indices plus FP16 values, so the bulk stays in dense low-bit form. The point is that the error contribution of a few weights is disproportionate to their count.

Paper's own claim (abstract): first near-lossless compression across model scales; relative perplexity loss under 1% for LLaMA and Falcon; runs a 33B-parameter LLM on a single 24GB consumer GPU with no performance degradation and a 15% speedup. COMPRESSION RATIO: 3-4 bits per weight PLUS the sparse outlier buffer, so the effective bits-per-weight is above the nominal 3-4 - the outliers are not free. QUALITY COST: <1% relative perplexity, which is the strongest quality-cost figure any 3-4 bit method in this KB reports.

arXiv ID 2306.03078 verified via /abs/. Venue ICLR 2024 from Semantic Scholar; year = venue year 2024 (v1 2023-06-05). The '24GB consumer GPU' claim is stated generically in the abstract; RTX 4090 is the 24GB part in this repo's accelerator set and is named as the hardware class the claim depends on. Venue was NOT cross-confirmed via arXiv comment field (which reads only 'Extended preprint'). arXiv ID and title verified by fetching https://arxiv.org/abs/2306.03078 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.03078) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.