LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

INT8 quantization of LLMs fails because a tiny set of emergent outlier feature dimensions dominates attention output and destroys the quantization error budget.

Two-part mixed-precision decomposition: (1) vector-wise quantization - each inner product in the matmul keeps its own normalization constant computed over the row of activations, instead of one per-tensor scale; (2) mixed-precision decomposition - outlier feature dimensions (columns with large systematic magnitude) are extracted into a small 16-bit matmul whose partial products are summed separately, so >99.9% of values still multiply in 8-bit. Keeps the outliers out of the INT8 GEMM entirely rather than widening it.

Paper's own claim (abstract): halves inference memory vs 16/32-bit while 'retaining full precision performance'; a 175B-parameter 16/32-bit checkpoint loads, converts to INT8 and runs with no performance degradation; >99.9% of multiplications occur in 8-bit. COMPRESSION RATIO: 2x weight memory (16-bit -> 8-bit). QUALITY COST: stated as zero ('without any performance degradation') - this is a self-reported claim against OPT/BLOOM zero-shot suites, not an independent measurement.

arXiv ID 2208.07339 verified via /abs/ page; v2. Venue: Semantic Scholar 'Advances in Neural Information Processing Systems 35'. Year = venue year 2022 (= v1 year). The 'retains full precision performance' claim is the paper's own and is a same-baseline self-comparison against the FP16 checkpoint. arXiv ID and title verified by fetching https://arxiv.org/abs/2208.07339 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2208.07339) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.