inference/research
verifiedhigh · 0.85updated 2026-10-03

vLLM docs: Disaggregated Prefilling (experimental)

repo

Fields

id
sources/pd-vllm-disagg-prefill-docs
Publisher
vLLM project
Kind
repo
Accessed
2026-10-03
notes
States the motivation as latency control, not throughput: 'Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately', and 'Controlling tail ITL' because without disaggregation vLLM 'may insert some prefill jobs during the decoding of one request'. Explicitly states the counter-intuitive constraint: 'Disaggregated prefill DOES NOT improve throughput.' Also: 'Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value', which is the honest comparison against chunked prefill. Lists 9 connector types (ExampleConnector, LMCacheConnectorV1, NixlConnector, MooncakeConnector, MoRIIOConnector for ROCm, ECMooncakeConnector, MultiConnector, OffloadingConnector, FlexKVConnectorV1) and the three connector abstractions (Connector, LookupBuffer with insert/drop_select, Pipe with send_tensor/recv_tensor) plus scheduler-side and worker-side connector split.

Notes

States the motivation as latency control, not throughput: 'Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately', and 'Controlling tail ITL' because without disaggregation vLLM 'may insert some prefill jobs during the decoding of one request'. Explicitly states the counter-intuitive constraint: 'Disaggregated prefill DOES NOT improve throughput.' Also: 'Chunked prefill with a proper chunk size also can achieve the same goal, but in practice it's hard to figure out the correct chunk size value', which is the honest comparison against chunked prefill. Lists 9 connector types (ExampleConnector, LMCacheConnectorV1, NixlConnector, MooncakeConnector, MoRIIOConnector for ROCm, ECMooncakeConnector, MultiConnector, OffloadingConnector, FlexKVConnectorV1) and the three connector abstractions (Connector, LookupBuffer with insert/drop_select, Pipe with send_tensor/recv_tensor) plus scheduler-side and worker-side connector split.

Cross-references

This record references no other records.