Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Published at USENIX FAST 2025 (Best Paper Award per the project README) and in ACM Trans. Storage (doi 10.1145/3773772). Abstract states the three-tier claim: a 'KVCache-centric disaggregated architecture that separates the prefill and decoding clusters' plus 'a disaggregated cache of KVCache' built from 'the underutilized CPU, DRAM, and SSD resources of the GPU cluster'. Also the source for the production-scale operating regime: 'Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios', leading to a prediction-based early-rejection policy. Claims up to 525% throughput increase in simulated scenarios while adhering to SLOs, and 75% more requests handled under real workloads. Both PD-disaggregation and the CPU/DRAM/SSD KV tier are in one system, which is why this record covers both flop classes.