The Llama 3 Herd of Models

Post-training quality has been getting less attention than pre-training scale, and the distillation/rejection-sampling recipe is the least documented part of frontier pipelines.

The distillation link in the Llama 3 pipeline: after pre-training the 405B model, its OUTPUTS are used to post-train smaller Llama 3 models via rejection sampling and preference optimization, giving each size its own data and its own optimizer iteration counts rather than one global recipe. The paper also describes model pruning, distillation-based data curation (LLM-as-judge quality filters on pretraining data), post-training Quantization (FP8 training with BF16 for numerical stability), and scalable infrastructure - i.e. the whole post-training recipe in one document.

Paper's own claim (abstract): Llama 3 'delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks', with the largest model a dense 405B Transformer with a 128K context window. No per-model distillation delta is given in the abstract - the numeric per-size comparisons are in the paper's tables. The concrete quantization claim relevant to this KB is FP8 post-training quantization, which is a quality-preserving cast rather than a compression method.

arXiv ID 2407.21783 verified via /abs/; v1 2024-07-31. Venue: Semantic Scholar returns NO venue for this ID, so arxiv-preprint with year = v1 year 2024 - flagged in notes rather than assumed. adoption=in-production and open_weights=True: Llama 3 weights are released under the Llama community license. Recorded in this slice specifically for the DISTILLATION AND PRUNING sections of the post-training recipe, which are the parts other Llama 3 docs omit. arXiv ID and title verified by fetching https://arxiv.org/abs/2407.21783 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2407.21783) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.