Fast Inference from Transformers via Speculative Decoding

Fields

id

sources/flop-speculative-decoding

URL

https://arxiv.org/abs/2211.17192

Publisher

arXiv (Leviathan, Kalman, Matias; Google Research)

Kind

paper

Published

2022-11-30

Accessed

2026-10-03

notes

Draft model proposes k tokens cheaply, target model verifies all k in one parallel forward pass. Distribution-preserving, no retraining. Reports 2x-3x on T5-XXL vs the T5X implementation with identical outputs.

Notes

Draft model proposes k tokens cheaply, target model verifies all k in one parallel forward pass. Distribution-preserving, no retraining. Reports 2x-3x on T5-XXL vs the T5X implementation with identical outputs.

Cross-references

This record references no other records.