Accelerating Production LLMs with Combined Token/Embedding Speculators

Academic speculative-decoding papers optimise draft quality in isolation, but a production system must accelerate an ALREADY highly optimised base-model implementation, where the easy headroom is gone.

Draft models conditioned on BOTH the context vectors AND already-sampled tokens, trained specifically to predict high-quality N-GRAMS that the base model will then accept or reject. Conditioning on sampled tokens (not just context) is what lets a small speculator learn the repetitive structure that makes n-gram drafts acceptable. It is explicitly a production engineering report, not a new sampling algorithm - the verification is standard modified rejection sampling.

2x-3x wall-clock inference speedup of highly optimised base-model implementations. The load-bearing qualifier is 'highly optimised base model implementations' - the gain is measured against an already-tuned production serving stack, which is a much harder baseline than the plain-decoding baselines in the academic papers. As a technical report it does not disclose the exact model pair or acceptance rate in the abstract, so those are NOT recorded here rather than guessed.

ADOPTION: in-production - this describes speculators deployed in a production LLM serving environment (the author list and framing are that of a deployed service), so it is the only record in this speculative-decoding set whose adoption claim rests on actual production deployment rather than upstream engine support. VENUE NOTE: a self-described technical report with no venue; arXiv comment mentions only an upload date and added references, so venue = arxiv-preprint, year = ArXiv v1 (Apr 2024). INCLUDED DESPITE WEAKER REPORTING than its neighbours precisely because it is the most honest baseline in the set - a 2-3x gain over an already-optimised production stack is a harder-won number than 6.5x over plain autoregressive decoding, and the two are not comparable. Absence of an acceptance rate and model pair in the abstract is recorded as absence rather than filled in; that gap is a limitation of the source, not an omission here. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). The paper gives a GPU count but no memory size and no form factor: "All speculators are trained on 32 NVIDIA A100 gpus, with latent dimensionality 4096", and the latency table measures "per-token latency of greedy inference with a Llama2-7B model and our trained speculator on a single A100, in fp16 precision". Source: https://arxiv.org/pdf/2404.19124 . The only capacity statement is a negative one, "far beyond the capabilities of an A100 gpu, which we observed failing at b ~ 128 due to memory limits", which bounds nothing to 40 vs 80. Left dangling. AFFILIATIONS BACKFILLED 2026-10-04 (paper-fields pass, agent w3p): PDF first page states 'IBM Research' for all seven authors; corroborated by OpenAlex ('IBM (United States)'). Filled from the API/PDF rather than from memory, because an affiliation is a fact about a specific paper's author block and inference from the author's employer history is how wrong affiliations get shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.