Accelerating Large Language Model Decoding with Speculative Sampling

Same memory-bandwidth bottleneck as Leviathan et al.; addressed with an alternative to sequence-level drafting.

Sequence-level speculative sampling: a draft model proposes multiple token sequences (v1 uses independent sequences, v2 conditions the sequence on the previous via Input-level MCMC). Target verifies all candidates in one batched pass; rejection sampling with an independent-sample fallback preserves the exact target distribution within hardware numerics.

2x-2.5x decoding speedup for Chinchilla 70B in a DISTRIBUTED setup, versus no speculative sampling, with no sample-quality compromise and no modification to the model. Model pair is 4B draft -> 70B target; the 2-2.5x is a batch-1 latency figure.

VENUE NOTE: no venue is verifiable for this one - Semantic Scholar returns no venue and the arXiv comment field is empty, so venue is recorded as arxiv-preprint with year = arXiv v1 (Feb 2023). It is often cited alongside Leviathan et al. 2211.17192 and the two are independently conceived, contemporaneous, and NOT interchangeable: the arXiv comment on 2211.17192 explicitly cites this paper as concurrent work. Independent-publication nuance matters because acceptance-ratio and drafting-strategy numbers from one paper do not transfer to the other formulation. Code was released later than the paper; the arXiv comment field carries no venue claim. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). No A100 anywhere in the paper, and the experiments are not on GPUs at all: "We train a 4 billion parameter draft model optimised for sampling latency on 16 TPU v4s" and the reported "14.1ms/token" figure is "on 16 TPU v4s". Source: https://arxiv.org/pdf/2302.01318 . The paper's only hardware-generic language is "preserves the distribution of the target model within hardware numerics". Left dangling. AFFILIATIONS BACKFILLED 2026-10-04 (paper-fields pass, agent w3p): PDF first page: all seven authors carry the single footnote '1All authors from DeepMind'. This is the only affiliation statement on the page; OpenAlex returns none. Filled from the API/PDF rather than from memory, because an affiliation is a fact about a specific paper's author block and inference from the author's employer history is how wrong affiliations get shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.