MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

Conventional wisdom says speculative decoding only helps at SMALL batch sizes, so it is useless in the high-throughput serving regime that production actually runs in.

Analyses how the bottleneck SHIFTS as batch size and sequence length grow: at small batch decode is bandwidth-bound (speculation helps), but at large batch and long context the KV CACHE becomes the binding constraint. Attacks the KV bottleneck directly by running the DRAFT MODEL with a SPARSE KV CACHE, which scales with sequence length and batch size while the target keeps a dense cache. Adds a theoretical model that selects the optimal drafting strategy for a given configuration rather than a fixed one.

Up to 2.51x speedup for Llama3.1-8B at batch sizes 32 to 256 for moderate-to-long sequences, on various hardware and task types. This is the record that makes the batch-size caveat falsifiable: the batch sizes 32-256 are stated, and the gain is claimed precisely in the regime where the field had assumed speculation would not help.

ADOPTION: research-only - the sparse-KV draft-model idea is not in production engines. VENUE NOTE: no venue verifiable via Semantic Scholar, no arXiv comment venue, so venue = arxiv-preprint and year = ArXiv v1 (Aug 2024). This paper is the CORRECTIVE to every batch-1 speedup in this sub-area, and is the record to cite when someone quotes 2.7x-6.5x from Leviathan/EAGLE/EAGLE-3 without a batch size: at batch 1 that is real, and MagicDec shows a different and smaller gain mechanism is available at batch 32-256 where the batch-1 result does not transfer. The honest reading is that these are two different operating points, not competing claims. Also note 'various types of hardware' in the abstract - the per-hardware attribution is not recoverable from the abstract, so hardware_relevance lists accelerator families the class of work uses rather than a per-configuration attribution. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). The paper names A100 many times and gives a GPU count and tensor-parallel degree, but NO memory size and no form factor: "We conducted most experiments on 8 Nvidia A100 GPUs with 8-way Tensor Parallelism", "These experiments are conducted on 8xA100 GPUs", "We ran all these experiments on 8 Nvidia A100 GPU with 8-way Tensor Parallelism", "(budget=512 and gamma=2, 8x A100)". Source: https://arxiv.org/pdf/2408.11049 . Its Figures 3-5 plot A100 alongside L40, H100 and V100 on FLOPS-to-memory-bandwidth-ratio axes rather than capacity, and no spec figure is given. Left dangling: 40 GB and 80 GB are both fully consistent with everything the paper says. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'iclr', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'ICLR' paper page: ICLR 2025 poster 30513. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2408.11049). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). AFFILIATIONS BACKFILLED 2026-10-04 (paper-fields pass, agent w3p): OpenAlex institutions field. The paper's own title block attributes it to CMU. Filled from the API/PDF rather than from memory, because an affiliation is a fact about a specific paper's author block and inference from the author's employer history is how wrong affiliations get shipped. YEAR FIELD CORRECTED 2026-10-04 (same pass as the venue correction above): year 2024 -> 2025 to match the ICLR 2025 venue year. The prior value 2024 is the arXiv posting year and remains true of the preprint. status stays contested. This drift is why the venue correction and the year correction were done together rather than separately: setting venue without setting year leaves the record internally inconsistent. 2026-10-10: ICLR 2025 accepted titles index (idx3.py: https://iclr.cc/virtual/2025/papers.html?filter=titles) and poster page (idx5.py: https://iclr.cc/virtual/2025/poster/30513) confirm "MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding". Source pap-eff-magicdec notes cite arXiv preprint vs ICLR 2025 index. Sources disagree; status remains contested.