arXiv 2406.02532: SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

Fields

id

sources/w15-pap-specexec

URL

https://arxiv.org/abs/2406.02532

Publisher

arXiv

Kind

paper

Published

2024-06-04

Accessed

2026-10-11

notes

NeurIPS 2024 paper by Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, Max Ryabinin. Code at https://github.com/yandex-research/specexec. Introduces massively parallel speculative execution constructing cache trees of high-probability continuations from draft model, validated in a single pass of target model during RAM/SSD parameter offloading. Evaluates on RTX 4090, RTX 3090, RTX 4060, and A100.

Notes

NeurIPS 2024 paper by Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, Max Ryabinin. Code at https://github.com/yandex-research/specexec. Introduces massively parallel speculative execution constructing cache trees of high-probability continuations from draft model, validated in a single pass of target model during RAM/SSD parameter offloading. Evaluates on RTX 4090, RTX 3090, RTX 4060, and A100.

Cross-references

This record references no other records.