inference/research
verifiedhigh · 0.90updated 2026-10-04

Mooncake#4209: TCP transport queue-full after idle gap causes PD hang

forum

Fields

id
sources/src5-gotcha-mooncake-4209-tcp-queue-full
Publisher
Mooncake
Kind
forum
Published
2026-09-18
Accessed
2026-10-04
notes
TCP transport: first KV transfer after idle gap (~5s) fails with queue-full rejections. Producer rejects all slices, batch_transfer_sync_write returns -1. Consumer hangs until timeout. Self-heals after ~90s. Root cause: retired group not removed from state->groups. Demonstrates multi-GPU silent failure in KV transfer.

Notes

TCP transport: first KV transfer after idle gap (~5s) fails with queue-full rejections. Producer rejects all slices, batch_transfer_sync_write returns -1. Consumer hangs until timeout. Self-heals after ~90s. Root cause: retired group not removed from state->groups. Demonstrates multi-GPU silent failure in KV transfer.

Cross-references

This record references no other records.