Speculative decoding that works standalone crashes under PD disaggregation, on a per-algorithm basis with no warning
[major] framework
Fields
id
gotchas/pd-speculative-decoding-unsupported-per-algorithm-under-pdClass
framework
Severity
major
Symptom
DFLASH speculative decoding crashes at startup under PD disaggregation: the first update_running_batch on the decode side dies with AttributeError: 'NoneType' object has no attribute 'prepare_for_decode' at sglang/srt/speculative/spec_utils.py:1010. Once that is locally patched, a second crash follows on real requests, ending in a watchdog self-kill. DSPARK and EAGLE both work under PD with the same deployment.
Root cause
PD support for speculative decoding is implemented per algorithm, not generically. SpeculativeAlgorithm.build_disagg_draft_input() implements EAGLE and DSPARK but returns None for DFLASH, so DisaggDecodeScheduleBatchMixin never sets batch.spec_info for relayed requests - a request that arrived via the prefill handoff has no speculative state. A contributing factor: the PD-decode event loop also runs update_running_batch on IDLE batches, which never go through build_disagg_draft_input at all, so even a correct implementation would be bypassed there.
Workaround
Check per-algorithm PD support before enabling speculative decoding on the decode pool - the presence of spec decode in a working homogeneous deployment says nothing about disaggregated support. Guard the decode path against a null spec_info on idle batches. Local fixes described in the report: add build_dflash_disagg_draft_input() on make_draft_input_v2(bonus_tokens=..., new_seq_lens=...) mirroring build_dspark_disagg_draft_input, and a create_idle_input() fallback when spec_info is None.
notes
Env: lmsysorg/sglang:kimi-k3 (0.5.16-based, also reproducible on latest main), Kimi-K3 + Kimi-K3-DFlash draft, 8x RTX 6000D (sm_120) across 4 prefill + 4 decode nodes, KV transfer Mooncake RDMA 0.3.12.post1. Open at 2026-10-03. The structural point: disaggregation adds a relay path into the decode engine, and every piece of per-request state on that path has to be reconstructed from the handoff payload. Anything with a per-algorithm builder is a place where that reconstruction can be missing. Dynamo enforces the analogous precondition structurally - require_prefill_result() raises on a decode-mode worker handed a request with no prefill_result - which is the pattern to copy.
Notes
Env: lmsysorg/sglang:kimi-k3 (0.5.16-based, also reproducible on latest main), Kimi-K3 + Kimi-K3-DFlash draft, 8x RTX 6000D (sm_120) across 4 prefill + 4 decode nodes, KV transfer Mooncake RDMA 0.3.12.post1. Open at 2026-10-03. The structural point: disaggregation adds a relay path into the decode engine, and every piece of per-request state on that path has to be reconstructed from the handoff payload. Anything with a per-algorithm builder is a place where that reconstruction can be missing. Dynamo enforces the analogous precondition structurally - require_prefill_result() raises on a decode-mode worker handed a request with no prefill_result - which is the pattern to copy.