Fast Inference from Transformers via Speculative Decoding

Autoregressive decoding is memory-bandwidth-bound: each new token requires a full pass over model weights, so decode FLOPs are wasted and accelerator compute sits idle.

Two-model draft-verify. A small draft model autoregressively proposes K tokens; the large target model scores all K+1 positions in ONE forward pass using custom attention masks (each position attends to its own prefix of draft tokens). A modified rejection-sampling acceptance rule (accept with min(1, p_target(x)/q_draft(x))) plus resampling from the target's residual distribution makes the output distribution provably identical to plain autoregressive decoding. Requires no target-model modification.

2x-3x wall-clock speedup at temperature 0 and up to ~2.5x with temperature sampling, on T5-XXL / T5-XXL-Target (T5-11B) and LLaMA-65B, at batch size 1-64. Acceptance-rate-per-step (tau) is the number that makes the speedup meaningful; the paper reports mean acceptance length as a function of draft K and model pair. Baseline is unmodified autoregressive decoding of the same target model.

THE canonical speculative-decoding record: ICLR 2023, the origin of the draft-model-plus-modified-rejection-sampling formulation everything else (SpecInfer, Sequoia, EAGLE, Medusa) reuses. VENUE NOTE: venue/year are taken from the arXiv comment field ('ICLR 2023') cross-checked against Semantic Scholar; arXiv v1 is Nov 2022, ICLR 2023 is the venue year. Adoption is 'in-production' because the ALGORITHM (not the code) is shipped: TensorRT-LLM, vLLM (n-gram + model spec decode), llama.cpp (draft-model spec decode), and SGLang all implement draft-verify with modified rejection sampling. Hard caveat for anyone benchmarking this: at large batch size the verify pass becomes compute-bound rather than bandwidth-bound and the speedup collapses - this is exactly what MagicDec (pap-eff-magicdec) and FlashInfer-style batch-size-aware work attack. A speedup quoted without batch size and without the acceptance rate is meaningless for this technique. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). No A100 and no GPU named anywhere in the paper; the experiments are TPU-based ("We demonstrate it on T5-XXL and show a 2X-3X acceleration compared to the standard T5X implementation", and the implementation "compare it to the implementation in the T5X codebase"). Source: https://arxiv.org/pdf/2211.17192 . Its one hardware sentence is generic: "the value of c depends on the hardware configuration and software implementation details". Left dangling. VENUE CORRECTED 2026-10-04 (paper-verification pass): venue/year iclr 2023 -> icml 2023. EVIDENCE: Title found in PMLR v202 (ICML 2023) at score 1.0; absent from the ICLR 2023 accepted-paper index (https://iclr.cc/virtual/2023/papers.html?filter=titles). Semantic Scholar independently returns venue "International Conference on Machine Learning" year 2022 (v1 arXiv year). This is THE canonical speculative-decoding record and its note explicitly claims "ICLR 2023, the origin of the draft-model-plus-modified-rejection-sampling formulation". Widely cited as an ICLR 2023 paper; the publication venue is ICML 2023. The ICLR claim was sourced to the arXiv comment field, which reads "ICLR 2023" — so either the authors mislabelled it or the comment is stale; either way the proceedings are authoritative and this is a peer-reviewed attribution, so it is flagged contested rather than silently rewritten. Both values retained here deliberately; status set to contested because a peer-reviewed attribution changed. arxiv_id unchanged (2211.17192). AFFILIATIONS FILLED 2026-10-04 (paper-verification pass): from the Semantic Scholar graph API (paper/arXiv:2211.17192), which is the only machine-readable source that carries author affiliations — the arXiv Atom feed returns author names only, and no ORCID is exposed by either API. One known upstream typo corrected: "Tel Aviv Univesity" -> "Tel Aviv University". S2 affiliation coverage is partial (120 of 249 corpus records carry any), so these lists may be incomplete rather than exhaustive. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. 2026-10-10: removed nvidia-a100-sxm and nvidia-h100-sxm. The experiments are TPU-based (T5-XXL, T5X implementation); no NVIDIA GPU is named anywhere. Both refs were fabricated. There is no TPU accelerator record in this repo, so the list is empty rather than pointing elsewhere. 2026-10-10: removed nvidia-a100-sxm and nvidia-h100-sxm. The experiments are TPU-based (T5-XXL, T5X implementation); no NVIDIA GPU is named anywhere. Both refs were fabricated. There is no TPU accelerator record in this repo, so the list is empty rather than pointing elsewhere.