Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

Training a competitive 1-3B model from scratch on trillions of tokens is expensive; is there a cheaper route to a small capable model than pretraining?

TARGETED structured pruning to an exact target SHAPE (remove whole layers, attention heads, and intermediate/hidden dimensions end-to-end, keeping the residual architecture valid) rather than unstructured pruning, which keeps a dense-shaped model with no speed benefit. Paired with DYNAMIC BATCH LOADING: reweight the sampling of each training batch toward the domains with the highest current loss, which is what makes recovery from the pruning damage fast on a small compute budget.

Paper's own claim (abstract): Sheared-LLaMA 1.3B and 2.7B, pruned from LLaMA2-7B, outperform SOTA open models of equivalent size (Pythia, INCITE, OpenLLaMA, TinyLlama) across downstream and instruction-tuning evaluations, WHILE REQUIRING ONLY 3% OF THE COMPUTE compared to training such models from scratch. COMPRESSION RATIO: 7B -> 1.3B or 2.7B is 2.6x or 5.4x fewer parameters, all of it structural so the speedup is real. QUALITY COST: no perplexity delta in the abstract; the reported axis is 'beats same-size models from scratch at 3% of the compute'. Note this is a TRAINING-cost claim about producing a small model, not an inference-compression claim about serving a large one.

arXiv ID 2310.06694 verified via /abs/; venue ICLR 2024 confirmed by the arXiv comment field-adjacent metadata AND independently by an OpenReview search which returned ICLR.cc/2024/Conference for the exact title. Year = venue year 2024 (v1 2023-10-10). arXiv ID and title verified by fetching https://arxiv.org/abs/2310.06694 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2310.06694) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.