On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

Sequence-level KD suffers a train/inference mismatch: the student is trained on a FIXED set of output sequences but generates its own at inference, and the errors compound.

Generalized Knowledge Distillation (GKD): instead of training only on the fixed output set, train the student on its OWN generated sequences with teacher feedback on those sequences - so the teacher supervises exactly the states the student actually reaches. Because the loss is decoupled from the on-policy data, ANY student-teacher divergence can be substituted, which matters when the student lacks the expressivity to match the teacher's distribution. The same objective composes with RL fine-tuning, letting distillation and RLHF run as one loop.

Paper's own claim (abstract): GKD addresses the distribution mismatch by training on self-generated sequences with teacher feedback, offers flexibility to employ alternative loss functions 'when the student lacks the expressivity to mimic the teacher's distribution', and integrates seamlessly with RLHF; demonstrated on summarization, translation, arithmetic reasoning, and task-agnostic instruction-tuning distillation. No numeric accuracy delta in the abstract.

arXiv ID 2306.13649 verified via /abs/. NOTE: the arXiv TITLE on the /abs/ page is 'On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes', while Semantic Scholar /paper/search/match returns the SAME arXiv ID under the title 'GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models'. Both titles are genuinely in circulation for this paper; the arXiv page title is recorded here as authoritative and the S2 alias noted so a reader searching either string finds this record. Venue ICLR 2024 confirmed by the arXiv comment field ('Accepted at ICLR 2024') and Semantic Scholar; year = venue year 2024 (v1 2023-06-23). arXiv ID and title verified by fetching https://arxiv.org/abs/2306.13649 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.13649) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.