SpecExec (Massively Parallel Speculative Execution)
cuda
Fields
id
engines/niche-specexecRepo
yandex-research/specexec
Languages
Python
First release
2024
Design goal
speculative-decodinglow-latencyconsumer-gpu-offload
Scheduling
none
Backends
cuda
Formats
safetensorsgptq
Notable features
Massively parallel speculative decoding tailored for consumer hardware with host RAM/SSD offloadingLeverages spikiness of token probability distributions in modern LLMs: runs draft search to find B most likely future tokens and builds a speculative cache treeTarget model computes probabilities for all candidate tokens in a single offloaded forward pass, generating up to 10-20 accepted tokens per target model iterationSupports stochastic sampling with exact distribution preservation (lossless) via generalized rejection samplingDemonstrates interactive inference for 50B+ models (e.g. Llama-2-70B, Llama-3-70B, Mixtral-8x7B) at 4-6 tok/s in 4-bit GPTQ and 2-3 tok/s in FP16 on single consumer GPUs (RTX 4090, RTX 3090, RTX 4060)
Hardware caveats
Designed specifically for RAM/SSD parameter-offloading regimes where GPU compute is vastly underutilized relative to memory bandwidth / PCIe busOn non-offloaded datacenter GPUs (like A100/H100 fitting models in VRAM), speedup is substantially lower (~2.1-2.4x at batch 1) because drafting overhead becomes a bottleneckBatch size scaling: primary speedup is at single-stream interactive decode (batch 1); concurrent serving with high batch sizes dilutes the advantage as GPU compute fills upNo standard open-source license asserted in repository root (license is null)Research prototype implementation (test scripts run_exp.py); lacks continuous batching or OpenAI-compatible server daemon
Best for
Running 70B+ LLMs interactively on consumer GPUs (16GB-24GB VRAM) with RAM offloadingUnderstanding massive draft-tree construction (budgets of 256-2048 tokens) when target model forward pass latency is measured in secondsLossless speculative sampling on offloaded setups
Avoid for
High-throughput multi-user serving (continuous batching is absent)Datacenter servers where large models already fit fully in high-bandwidth HBM3/HBM3eProduction environments requiring OSI license compliance without legal review
Tracing support
none
Verified negatives
yes
notes
SpecExec (arXiv:2406.02532, NeurIPS 2024) turns the traditional speculative decoding logic on its head: while datacenter spec-decode budgets small draft chains (k=3-8) to avoid compute overhead, RAM offload shifts the bottleneck entirely to weight streaming over PCIe. Because streaming weights takes seconds, computing thousands of speculative draft tokens in the target model forward pass costs negligible extra time. Uses draft trees up to 2048 tokens to accept 10-20 tokens per pass.
Notes
SpecExec (arXiv:2406.02532, NeurIPS 2024) turns the traditional speculative decoding logic on its head: while datacenter spec-decode budgets small draft chains (k=3-8) to avoid compute overhead, RAM offload shifts the bottleneck entirely to weight streaming over PCIe. Because streaming weights takes seconds, computing thousands of speculative draft tokens in the target model forward pass costs negligible extra time. Uses draft trees up to 2048 tokens to accept 10-20 tokens per pass.
Cross-references
This record references no other records — it is a leaf. It is cited by 1.