Languages
C++PythonCUDAGoRust
Design goal
throughput-firstlow-latencyKV-cache-tieringproduction-scale overload handling
Scheduling
continuous-batchingpaged-kvprefix-cachedisaggregated-prefill-decodeKVCache-centric scheduler
Notable features
Transfer Engine (TE): a unified batched-data-movement interface across storage, network and accelerator environments, with 'multiple transport protocols, topology-aware routing, multi-NIC bandwidth aggregation, and automatic failover'Mooncake Store: a distributed KV cache (and model-weight store) built on the Transfer EngineKVCache-centric disaggregated architecture separating prefill and decode clusters, plus a disaggregated KV cache pool built from otherwise-idle CPU, DRAM and SSD in the GPU clusterKVCache-centric scheduler that maximises effective throughput subject to latency SLOs, rather than assuming all requests are servedPrediction-based early-rejection policy for overloaded conditions - an explicit admission-control feature rather than an implicit oneThe broadest transport matrix in the ecosystem: tcp, rdma, efa (AWS libfabric SRD), nvmeof, nvlink (MNNVL inter-node), nvlink_intra, hip, barex, cxl, shm, musa (MTLink), ascend, hylink, tpu, mpcomm, flagcxWon USENIX FAST 2025 Best Paper; published in ACM Trans. Storage (doi 10.1145/3773772)
Hardware caveats
The TCP path is not merely a slower RDMA - it has its own failure mode: the first transfer after an idle gap fails with lane queue-full rejections and the decode side hangs until timeout (Mooncake #4209)Transport choice is deployment-specific and wrong choices are silent: auto-selected GPU-affine NICs on PCIe-passthrough VMs can pick a PIX-adjacent partner where GPUDirect RDMA does not work at allshm is documented as 'same-host DRAM copies without NIC loopback' - useful but not a GPU-to-GPU pathNon-CUDA support exists as separate PyPI wheels (ROCm, NPU, MUSA, EFA, non-CUDA), so accelerator coverage is real but packaged per target
Best for
Long-context production serving where the KV working set exceeds HBM and the idle CPU/DRAM/SSD in the cluster can be used as a real KV tierOverloaded deployments that need admission control rather than unbounded queueingHeterogeneous fabrics - the transport matrix (RDMA, EFA, NVLink, CXL, shared memory, vendor-specific fabrics) is unmatchedServing stacks that want a KV tier that outlives the engine process (the Store is a separate service)
Avoid for
Latency-sensitive single-request serving where a KV hop is pure overheadHosts without an RDMA-capable NIC, if you intend to run the RDMA protocol - TCP works but carries the queue-full hangSmall clusters with no idle DRAM/SSD to contribute - the disaggregated cache pool is the main differentiator and it needs spare capacity
notes
Research prototype or production system? Production. This is the serving platform for Kimi, with a FAST'25 Best Paper and an ACM Trans. Storage journal version, and the README documents production integration into vLLM (2024-12-16), SGLang (2025-04-10) and others. KV transfer path: the Transfer Engine, whose protocol is chosen per deployment - RDMA verbs over an HCA for inter-node, EFA/libfabric SRD on AWS, nvlink (MNNVL) for inter-node GPU-to-GPU, nvlink_intra for intra-node NVLink, hip for AMD, shm for same-host DRAM, plus nvmeof/cxl for deeper tiers. It is simultaneously a PD-disaggregation system and a KV-tier system, which is why it sources both flop records. Reported results: up to 525% throughput increase in certain simulated scenarios while adhering to SLOs, and 75% more requests handled under real workloads - both from the paper abstract, so treat them as the authors' own figures. Adoption surface is broad enough to be a compatibility target rather than just a dependency: it is an LMCache storage/transport backend, an SGLang transfer backend, and a vLLM connector family (MooncakeConnector, MooncakeStoreConnector, ECMooncakeConnector). It also maintains a public known-issues tracker (#2647). ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-30 (0.1 months before this record), latest release v0.3.13.post1 published 2026-08-31, 359 contributors (distinct committers including anonymous), open_issues_count 681 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 6713 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.