EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

EAGLE-2 introduces context-aware dynamic draft trees. Key measured results: speedup 3.05x-4.26x across Vicuna/LLaMA2/LLaMA3 on 6 tasks. Average acceptance length 4.2-5.5 tokens per draft-verify cycle (vs 2.4-2.6 for Medusa, 1.6-1.7 for PLD/Lookahead). EAGLE-2 is 20-40% faster than EAGLE-1. Claims 'lossless acceleration algorithm' - distribution unchanged. Draft tree depth 6, 10 nodes expanded per layer, 48-60 total draft tokens. Well-calibrated draft model: confidence score approximates acceptance rate (confidence >0.95 → acceptance ~0.98; confidence <0.05 → acceptance ~0.04). Accepted at ICML 2024.