inference/research
verifiedhigh · 0.85majorupdated 2026-10-03

EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger

[major] config

Fields

id
gotchas/eagle-prefix-cache-last-block-drop
Class
config
Severity
major
Concepts
hybrid-gdn-mambaprefix-cachingspeculative-decoding
Symptom
Multi-round and repeated-prompt workloads lose ~6-10 points of prefix-cache hit rate versus an identical no-spec boot (97.5% -> 86.9-91.7%). On one hybrid Qwen3.8 GDN layout each hit recomputes a 1,648-token scheduler-aligned unit, costing ~30-40% batch throughput on prefix-reusing workloads.
Root cause
The EAGLE last-block drop (drop_eagle_block) in the prefix-cache hit path. kv_cache_coordinator.py flags ALL KV groups as eagle when the model has no is_eagle_group annotation (a conservative fallback). The EAGLE-marked dense/full-attention group then drops one alignment unit from its hit - protecting against draft-layer KV for the last block depending on the writer's continuation, and against prefill-lookahead pollution at chunk boundaries. Hybrid reconciliation limits the combined hit to that shorter boundary. On dense-attention models the alignment unit is a 16-token block; on this hybrid GDN/Mamba layout it is 1,648 tokens, so the same protection costs ~100x more cache hit.
Workaround
Measure prefix-cache hit rate with and without speculative_config before enabling spec decoding on prefix-heavy workloads. Annotate is_eagle_group per group where the model allows it, or reduce scheduler alignment granularity on hybrid layouts.
notes
The 100x figure is layout-specific and the reporter explicitly says other hybrid layouts must be evaluated by their own scheduler alignment - do not generalize it. A separate upstream request (vLLM issue #53786) asks for fine-grained prefix hits for sliding-window groups, which would bear on the same hit-rate accounting.

Notes

The 100x figure is layout-specific and the reporter explicitly says other hybrid layouts must be evaluated by their own scheduler alignment - do not generalize it. A separate upstream request (vLLM issue #53786) asks for fine-grained prefix hits for sliding-window groups, which would bear on the same hit-rate accounting.

Cross-references