Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Fine-tuning and distilling small models both need large amounts of training data to match a prompted LLM.

Extract the LLM's RATIONALES as additional supervision and train the small model on (input, rationale, answer) triples inside a multi-task framework - the rationale is a training signal the student gets for free from the teacher. The result is that data EFFICIENCY, not model size, becomes the binding constraint: the student needs far fewer examples because each example now carries a reasoning trace.

Paper's own claim (abstract): across 4 NLP benchmarks, (1) better performance than both fine-tuning and distillation with far fewer training examples; (2) better performance than FEW-SHOT PROMPTED LLMs at substantially smaller model sizes; (3) a finetuned 770M T5 OUTPERFORMS THE FEW-SHOT PROMPTED 540B PaLM MODEL using only 80% of the available data on a benchmark, where standard finetuning of the same T5 'struggles to match even by using 100% of the dataset'. The 770M-beats-540B-on-80%-of-the-data figure is the headline and the 100%-failure of the baseline is what makes it meaningful.

arXiv ID 2305.02301 verified via /abs/; venue confirmed by the arXiv comment field ('Accepted to Findings of ACL 2023') and Semantic Scholar ('Annual Meeting of the Association for Computational Linguistics'). Year = venue year 2023 (Findings track, ACL 2023). Note the baseline is FEW-SHOT PROMPTED PaLM, not a trained 540B checkpoint - the comparison is against the prompting regime, which is the honest framing. arXiv ID and title verified by fetching https://arxiv.org/abs/2305.02301 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2305.02301) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'findings' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2023.findings-acl.507, booktitle 'Findings of the Association for Computational Linguistics: ACL 2023'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which.