The LLM Surgeon

Data-driven compression of pretrained LLMs usually treats pruning and weight-update as separate problems, and unstructured pruning of a large model was thought to need full second-order computation.

Scales Kronecker-FACTORIZED curvature approximations (KFAC) of the target loss landscape to LLM scale, which is what makes it affordable, and uses them to do two coupled things at once: dynamically allocate which structures can be removed, AND update the surviving weights to account for the removal. The weight-update term is improved to capture more correlations between weights than prior second-order methods. Unifies unstructured, semi-structured and structured pruning in one framework.

Paper's own claim (abstract): prunes ROWS AND COLUMNS from a range of OPT models and LLaMA2-7B BY 20%-30% with 'a negligible loss in performance', and achieves state-of-the-art results in unstructured and semi-structured pruning of LLMs. COMPRESSION RATIO: 20-30% structural removal (1.25x-1.43x), and higher when unstructured. QUALITY COST: 'negligible loss in performance' with no paired number in the abstract. Note the tension with the training-free methods above: LLM Surgeon DOES update weights, which is where its quality comes from and what makes it slower than Wanda.

arXiv ID 2312.17244 verified via /abs/; venue: Semantic Scholar returns arXiv.org only and an OpenReview search found no accepted-venue record for the exact title (only an unrelated 2026 workshop paper) - recorded as arxiv-preprint with year = v1 year 2023. arXiv ID and title verified by fetching https://arxiv.org/abs/2312.17244 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2312.17244) unless the record's notes say otherwise. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'iclr', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'ICLR' paper page: ICLR 2024 poster 19140. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2312.17244). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). YEAR FIELD CORRECTED 2026-10-04 (same pass as the venue correction above): year 2023 -> 2024 to match the ICLR 2024 venue year. The prior value 2023 is the arXiv posting year and remains true of the preprint. status stays contested. This drift is why the venue correction and the year correction were done together rather than separately: setting venue without setting year leaves the record internally inconsistent. 2026-10-10: ICLR 2024 poster page (idx5.py: https://iclr.cc/virtual/2024/poster/19140) and papers index (idx6.py: https://iclr.cc/virtual/2024/papers.html?filter=titles) confirm "The LLM Surgeon". arXiv API 2312.17244 (axc.py) verified. Source pap-quant-llm-surgeon notes cite Semantic Scholar vs ICLR 2024 index. Sources disagree; status remains contested.