vLLM issue #56828: KV Offload P2P peer-down on one rank does not fail loads or block new loads to sibling ranks
Fields
sources/pd-gh-vllm-56828-p2p-peer-down-no-failNotes
Disaggregated P/D deployment (llm-d), multi-rank prefill pods with 8 P2P control ports. When a source pod died mid-transfer, the consumer marked only that rank's peer down and then submitted a NEW load to a sibling rank of the same dead pod, which ran the full 30s load timeout; requests bound to that source fell back to local prefill only after 36-43s. Confirmed by maintainer he-yufeng against main: peer_id is per control port so an 8-rank pod registers as 8 independent peers; _reap_dead_sessions only reaps the session whose own ZMQ connection died; there is no progress field anywhere in the load path, so 'the 30s walk is the only terminator'. Also: a rank that was still answering control messages while its pod failed health checks meant the ZMQ heartbeat never fired - the data path stalled on a control-alive peer.
Cross-references
This record references no other records.