llm-d Disaggregated Serving Operations (vLLM) - fault tolerance, KV leases, rollouts, known NIXL issues

Prefill failure: 'D instances will attempt to pull KVs from no longer running P instances without performing any liveness checks.' A failed READ returns an error such as NIXL_ERR_BACKEND and kv_load_failure_policy decides: 'fail (default, recommended): Immediately fail the request with an error when KV load fails. This prevents performance degradation by avoiding recomputation of prefill work on the decode instance.' versus 'recompute: Recompute failed blocks locally on the decode instance. This may cause performance jitter on decode instances as the scheduled prefill will delay and interfere with other decodes. Furthermore, decode instances configured with low-latency optimizations (such as DeepEP LL for Wide EP deployments) may suffer significant slowdowns.' Neither path re-routes or retries the request elsewhere. Decode failure strands KV on the prefill instance; vLLM answers with lease-based KV block management: initial lease kv_lease_duration default 30s, D sends heartbeats each extending the lease by kv_lease_duration * 2 // 3 (default 20s), and 'If D crashes (or becomes unresponsive), heartbeats stop, and P automatically frees the KV blocks when the last lease extension expires - at most lease_extension seconds after the last heartbeat.' Lease-based TTL requires vLLM v0.22.0 or later; earlier versions used a fixed VLLM_NIXL_ABORT_REQUEST_TIMEOUT default 480s, where 'the P instance would free stranded KV blocks only after that timeout elapsed, regardless of whether the D instance was still alive. This made the worst-case hold time very long, and reducing the timeout risked premature eviction when D instances were heavily loaded.' Scale-down of decode replicas: 'regardless of whether --shutdown-timeout is set, requests are aborted during the shutdown process', and decode instances with not-yet-started KV transfers notify remote prefill instances to free blocks. Rollouts: 'vLLM has multiple attention kernel implementations, each of which can have slightly different KV cache layouts - since NIXL pulls the KVs directly from the GPU KV cache memory of the remote instance, we need to ensure these are matching.' vLLM checks compatibility during the NIXL handshake and fails the request if scheduled to incompatible pods, with enforce_handshake_compat as the escape hatch; 'The llm-d EPP currently assumes all P and D instances within an InferencePool are compatible and will therefore schedule requests to any arbitrary pair of P and D instances. As a result, it is currently recommended to create a new InferencePool for upgrading model servers. When deploying with a Gateway, traffic can be gradually shifted to the new InferencePool by modifying the HTTPRoute.' Known limitation: prefill TP > decode TP is explicitly unsupported for non-MLA, non-Mamba models and 'A related bug in vLLM failure-cleanup handler (_handle_failed_transfer) often surfaces it as IndexError: list index out of range instead, making it look like a generic connector bug rather than an unsupported-topology guard.' Stale agent cache: after a prefill pod restart, a decode replica holding a cached NIXL agent handle 'can segfault, or leave it silently serving against a dead engine for up to an hour. Until the fixes below ship in an official image, proactively restart any decode replica that was connected to a restarted prefill pod, rather than relying on TTL expiry.' With a pinned engine ID 'decode does not detect the prefill engine is gone until engine_ttl expires (default up to one hour), and every request routed to that decode replica fails until then or until the decode pod is restarted.' Under recompute 'The client never sees a failed request, so this shows up purely as a throughput/latency regression on that decode replica, not as an error - harder to notice than an outright failure.' Fix status: NIXL-level segfault fixed in ai-dynamo/nixl#1986 (merged ~2026-07-29) and vLLM-side TTL invalidation in vllm-project/vllm#50047, but 'Neither fix is in any published vLLM image yet, including nightly, since vLLM pins an exact NIXL version and the NIXL fix landed after the latest published NIXL release (v1.3.2, 2026-07-24).'