inference/research
verifiedhigh · 0.85updated 2026-10-03

vLLM issue #52627: Kimi-K3 silent output corruption in 1P1D NIXL Direct-PD disaggregated deployment

forum

Fields

id
sources/pd-gh-vllm-52627-kimi-k3-pd-silent-corruption
Publisher
vllm-project/vllm
Kind
forum
Published
2026-08-17
Accessed
2026-10-03
notes
Open as of 2026-10-03. Intermittent SILENT output corruption in a two-node 1P1D PD deployment, Kimi-K3 TP=8, with MultiConnector stacking NixlConnector (kv_producer) and MooncakeStoreConnector (kv_both, load_async, lookup_async). The A/B isolation is the valuable part: 'NIXL-only PD: stable, no garbled output observed'; 'NIXL + MooncakeStoreConnector through MultiConnector: garbled output reproducibly appears under sustained load'; 'Mooncake enabled + DSpark/speculative decoding disabled: corruption still reproduces'; 'Mooncake disabled: corruption has not reproduced'. Symptom: long runs of nonsensical mixed-language/repeated tokens, some streams surfacing later as HTTP 502 / 7104_UPSTREAM_STREAM_INTERRUPTED, with the engine staying up - 'the engine can continue running while individual outputs are already corrupted'. So: stacking a KV-tier connector under the P/D transfer connector is a distinct failure mode from either alone.

Notes

Open as of 2026-10-03. Intermittent SILENT output corruption in a two-node 1P1D PD deployment, Kimi-K3 TP=8, with MultiConnector stacking NixlConnector (kv_producer) and MooncakeStoreConnector (kv_both, load_async, lookup_async). The A/B isolation is the valuable part: 'NIXL-only PD: stable, no garbled output observed'; 'NIXL + MooncakeStoreConnector through MultiConnector: garbled output reproducibly appears under sustained load'; 'Mooncake enabled + DSpark/speculative decoding disabled: corruption still reproduces'; 'Mooncake disabled: corruption has not reproduced'. Symptom: long runs of nonsensical mixed-language/repeated tokens, some streams surfacing later as HTTP 502 / 7104_UPSTREAM_STREAM_INTERRUPTED, with the engine staying up - 'the engine can continue running while individual outputs are already corrupted'. So: stacking a KV-tier connector under the P/D transfer connector is a distinct failure mode from either alone.

Cross-references

This record references no other records.