Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Models distilled from large foundation models tend to learn the ANSWER STYLE of the teacher rather than its REASONING PROCESS, and the small model's capability is then systematically overestimated because evaluation does not catch this.

Imitation of the reasoning process rather than the output, using three richer signal types from GPT-4 - explanation traces, step-by-step thought processes, and complex instructions - guided by ChatGPT as teacher assistant, sampled from large-scale diverse imitation data with judicious sampling and selection to promote PROGRESSIVE learning (each stage harder than the last) instead of flat imitation. The paper is explicit that the released artifact is a 13B model trained by Microsoft.

Paper's own claim (abstract): Orca (13B) 'surpasses conventional state-of-the-art instruction-tuned models such as Vicuna-13B by MORE THAN 100% IN COMPLEX ZERO-SHOT REASONING' benchmarks. The 'more than 100%' is a relative-improvement figure (roughly 2x the benchmark score of Vicuna-13B), NOT a 100-point improvement - the phrasing in the paper is easily misread and this record quotes it as the paper does while flagging the reading. COMPRESSION RATIO: 13B student against a much larger GPT-4 teacher. QUALITY COST: the paper's own thesis is that this COST was invisible under the prevailing evaluation regime, which is the most useful thing it says.

arXiv ID 2306.02707 verified via /abs/; venue: Semantic Scholar returns arXiv.org only and the arXiv comment field is empty - arxiv-preprint, year = v1 year 2023. adoption=in-production: Orca was the first Llama-based model shipped in a Microsoft production service (Azure OpenAI), which is a deployment fact about the artifact, not about this paper's recipe. arXiv ID and title verified by fetching https://arxiv.org/abs/2306.02707 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.02707) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.