LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

Speculative decoding needs a separate draft model, doubling model memory; and early-exit inference needs auxiliary exit heads, which cost accuracy and add modules.

TRAINING RECIPE, not an architecture. Applies LAYER DROPOUT during training with LOW rates for EARLY layers and HIGH rates for LATE layers, plus an EARLY EXIT LOSS in which all transformer layers share the SAME exit. At inference, exit early and use the REMAINING LAYERS of the same model to VERIFY AND CORRECT the early exit's draft - self-speculative decoding with the model as its own drafter and verifier. No auxiliary layers or modules are added, so memory footprint is lower than approaches needing a distinct draft model, and draft and verify stages SHARE activations and compute.

Speedups of up to 2.16x on summarization of CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing, across different Llama sizes and across four training regimes (pretraining from scratch, continual pretraining, domain finetuning, task finetuning). Model pairs are same-model (early layers draft, later layers verify) with Llama checkpoints.

ADOPTION, precisely: code AND checkpoints are open (facebookresearch/LayerSkip) but the method is NOT in production engines as a default - llama.cpp has related self-speculative/early-exit paths and vLLM has no shipped LayerSkip path, so research-only with released artifacts is the accurate label. Its DISTINCTIVE claim versus all other self-speculative records here is the MEMORY one: draft and verify share weights, so there is no second model resident in HBM. That makes it the right technique when you are memory-bound rather than bandwidth-bound. VENUE: ACL 2024, corroborated twice - arXiv comment says 'ACL 2024' and there is a DOI 10.18653/v1/2024.acl-long.681 in the ACL Anthology namespace - so venue confidence is high. REPRODUCTION CAVEAT: this method REQUIRES a model trained (or continued-pretrained) with the layer-dropout schedule. It is not a post-hoc transform you can apply to an arbitrary checkpoint, so its speedups cannot be reproduced on models lacking that training - a different and stronger dependency than EAGLE or Medusa-1, which can be trained separately for any checkpoint. DANGLING REF RESOLVED 2026-10-04 by reading the paper (arXiv:2404.16710v1, Appendix A.1, Table 8 'Training Hyperparameters and Configuration of Experiments'). The GPU column reads "64 A100 80 GB" for Continual Pretraining of Llama2 7B and Llama2 13B, "32 A100 80 GB" for Finetuning on Code Data, and "8 A100 80 GB" for Finetuning on Task-Specific Dataset. Section 5 also says "The experiments were performed on NVIDIA A100 GPUs" for the code-finetuning runs. Source: https://arxiv.org/pdf/2404.16710 . 80 GB is stated, so this points at the 80 GB part. CAVEAT the table's own inconsistency, do not read it as a clean spec: two rows say 'A100 30 GB', which is not an A100 SKU that exists and is almost certainly a typo for 40 GB. The rows relevant to this record's speedup claims (code data, task-specific) are all 80 GB. AFFILIATIONS FILLED 2026-10-04 (paper-verification pass): from the Semantic Scholar graph API (paper/arXiv:2404.16710), which is the only machine-readable source that carries author affiliations — the arXiv Atom feed returns author names only, and no ORCID is exposed by either API. One known upstream typo corrected: "Tel Aviv Univesity" -> "Tel Aviv University". S2 affiliation coverage is partial (120 of 249 corpus records carry any), so these lists may be incomplete rather than exhaustive. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'main' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2024.acl-long.681, booktitle 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which.