A Simple and Effective Pruning Approach for Large Language Models

Magnitude pruning underperforms on LLMs, while second-order pruning methods need weight reconstruction that is expensive at billion-parameter scale.

Wanda (Pruning by Weights and Activations): score each weight by |W_ij| * ||X_j||_2, where X_j is the INPUT ACTIVATION norm of channel j, and drop the lowest-scoring weights per output row. The score is computed directly from the pretrained weights and a single small calibration set of activations - NO retraining, NO weight update, NO gradient. The pruned model is used as-is. The metric is also invertible, so Wanda can be applied without access to the original weights (privacy-preserving transfer).

Paper's own claim (abstract): evaluated on LLaMA and LLaMA-2 across various language benchmarks, Wanda 'significantly outperforms the established baseline of magnitude pruning' and 'performs competitively against recent method involving intensive weight update' - i.e. it matches SparseGPT-class methods while being training-free and gradient-free. COMPRESSION RATIO: unstructured sparsity at the level swept in the paper (up to ~50% typical for LLaMA-7B). QUALITY COST: reported as beating magnitude pruning with no quantitative delta in the abstract. Note again: UNSTRUCTURED sparsity yields memory savings but no acceleration on any current accelerator.

arXiv ID 2306.11695 VERIFIED by fetching /abs/2306.11695 - the title on the page is 'A Simple and Effective Pruning Approach for Large Language Models' (no 'Wanda' in the title); venue ICLR 2024 confirmed by Semantic Scholar (DBLP conf/iclr/Sun0BK24). Year = venue year 2024 (v1 2023-06-01). The common citation form 'Wanda: A Simple and Effective Pruning Approach...' has a title the arXiv record does not use. arXiv ID and title verified by fetching https://arxiv.org/abs/2306.11695 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.11695) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.