inference/research
verifiedhigh · 0.85updated 2026-10-03

SGLang docs: PD Disaggregation

repo

Fields

id
sources/pd-sglang-pd-disagg-docs
Publisher
sgl-project (SGLang)
Kind
repo
Accessed
2026-10-03
notes
States the two failure modes disaggregation removes: 'Prefill Interruption: Incoming prefill batches frequently interrupt ongoing decode batches' and 'DP Attention Imbalance: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency'. Transfer path: 'Currently, we support Mooncake and NIXL as the transfer engine', with per-instance flags such as --disaggregation-ib-device selecting specific RDMA HCAs, and an 'ascend' transfer backend for NPUs. Scheduling detail worth recording: --disaggregation-decode-allocation-policy prefill_complete defers decode-side KV slot reservation until prefill signals readiness, because 'By default, decode reserves each request's destination KV slots before prefill finishes. Long prefills can hold these slots while decode waits for the KV data.'

Notes

States the two failure modes disaggregation removes: 'Prefill Interruption: Incoming prefill batches frequently interrupt ongoing decode batches' and 'DP Attention Imbalance: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency'. Transfer path: 'Currently, we support Mooncake and NIXL as the transfer engine', with per-instance flags such as --disaggregation-ib-device selecting specific RDMA HCAs, and an 'ascend' transfer backend for NPUs. Scheduling detail worth recording: --disaggregation-decode-allocation-policy prefill_complete defers decode-side KV slot reservation until prefill signals readiness, because 'By default, decode reserves each request's destination KV slots before prefill finishes. Long prefills can hold these slots while decode waits for the KV data.'

Cross-references

This record references no other records.