Accelerating Large Language Model Decoding with Speculative Sampling

Fields

id

sources/flop-speculative-sampling-deepmind

URL

https://arxiv.org/abs/2302.01318

Publisher

arXiv (Chen et al.; DeepMind)

Kind

paper

Published

2023-02-02

Accessed

2026-10-03

notes

Independent concurrent formulation. Key claim: the latency of parallel scoring a short draft continuation is comparable to sampling one token from the target model - i.e. the draft is bandwidth-cheap while the verify step is compute-cheap, so they overlap well. 2-2.5x on Chinchilla 70B in a distributed setup.

Notes

Independent concurrent formulation. Key claim: the latency of parallel scoring a short draft continuation is comparable to sampling one token from the target model - i.e. the draft is bandwidth-cheap while the verify step is compute-cheap, so they overlap well. 2-2.5x on Chinchilla 70B in a distributed setup.

Cross-references

This record references no other records.