Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

Existing speculative-decoding methods cannot scale to large speculation budgets and do not adapt to differing hyperparameters or to different hardware.

Three components. (1) A DYNAMIC PROGRAMMING algorithm finds the optimal token TREE STRUCTURE for a given speculation budget, making tree shape optimal rather than hand-designed. (2) A novel sampling-and-VERIFICATION method that is robust across decoding TEMPERATURES - most prior work is tuned for greedy/low temperature and degrades badly at high temperature, which is a genuine production gap. (3) A HARDWARE-AWARE tree OPTIMIZER that automatically selects tree size and depth for the specific hardware platform, by measuring hardware characteristics rather than assuming them.

Decoding speedup up to 4.04x (Llama2-7B), 3.73x (Llama2-13B), and 2.27x (Vicuna-33B) on an A100. Separately, in the OFFLOADING setting on L40, 0.56 s/token for exact Llama2-70B inference, which the authors report as 9.96x over their own optimized offloading system (5.6 s/token), 9.7x over DeepSpeed-Zero-Inference, and 19.5x over HuggingFace Accelerate. The L40 offloading figure is the interesting one and is the reason this paper is in the repo: it is a per-token LATENCY figure in seconds, not a unitless multiplier, so it is directly comparable to the repo's benchmark records.

ADOPTION: research-only - the hardware-aware tree optimiser is not in any production engine, and the temperature-robust verification algorithm is not adopted upstream. VENUE NOTE: no venue verifiable via Semantic Scholar; the arXiv comment field carries no venue claim, so venue = arxiv-preprint, year = ArXiv v1 (Feb 2024). CAVEATS A READER MUST CARRY: the 9.96x is against the AUTHORS' OWN optimized offloading system, so it measures their tree optimiser against their own stack, not against the absence of speculative decoding - exactly the trap the xwind record in this repo documents for a different paper. The 4.04x/3.73x/2.27x series also declines monotonically with model size, which is the honest shape of the result: bigger target models amortise the draft pass better. Sequoia is the only record here where hardware_relevance is load-bearing in the mechanism itself rather than merely in the evaluation. DANGLING REF RESOLVED 2026-10-04 by reading the paper (arXiv:2402.12374v1), and it CORRECTS THE OLD SHORTHAND'S FORM FACTOR. The setup section names the exact part: "We evaluate SEQUOIA on different hardware including on-device experiments on L40 and A100(-PCIE 80GB) GPUs, as well as offloading experiments on an L40 GPU (with PCIE4)". Source: https://arxiv.org/pdf/2402.12374 . The paper's on-device results that this record's speedup claim rests on are the A100 runs (Table 1 'On-device results (A100)', Figure 7, Figure 8 forward-pass timings). So the card is the 80 GB PCIE, i.e. nvidia-a100-80gb-pcie, not the SXM4 part the old shorthand implied. AFFILIATIONS FILLED 2026-10-04 (paper-verification pass): from the Semantic Scholar graph API (paper/arXiv:2402.12374), which is the only machine-readable source that carries author affiliations — the arXiv Atom feed returns author names only, and no ORCID is exposed by either API. One known upstream typo corrected: "Tel Aviv Univesity" -> "Tel Aviv University". S2 affiliation coverage is partial (120 of 249 corpus records carry any), so these lists may be incomplete rather than exhaustive. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.