vLLM issue #52627: Kimi-K3 silent output corruption in 1P1D NIXL Direct-PD disaggregated deployment
Fields
sources/pd-gh-vllm-52627-kimi-k3-pd-silent-corruptionNotes
Open as of 2026-10-03. Intermittent SILENT output corruption in a two-node 1P1D PD deployment, Kimi-K3 TP=8, with MultiConnector stacking NixlConnector (kv_producer) and MooncakeStoreConnector (kv_both, load_async, lookup_async). The A/B isolation is the valuable part: 'NIXL-only PD: stable, no garbled output observed'; 'NIXL + MooncakeStoreConnector through MultiConnector: garbled output reproducibly appears under sustained load'; 'Mooncake enabled + DSpark/speculative decoding disabled: corruption still reproduces'; 'Mooncake disabled: corruption has not reproduced'. Symptom: long runs of nonsensical mixed-language/repeated tokens, some streams surfacing later as HTTP 502 / 7104_UPSTREAM_STREAM_INTERRUPTED, with the engine staying up - 'the engine can continue running while individual outputs are already corrupted'. So: stacking a KV-tier connector under the P/D transfer connector is a distinct failure mode from either alone.
Cross-references
This record references no other records.