Design goal
low-latencyTTFT-reductionvendor-neutral KV reuseKV-tiering
Scheduling
prefix-cacheradix-cachedisaggregated-prefill-decodeKV-tiering
Notable features
A KV cache management layer, not a GPU kernel library - it turns KV from temporary state into persistent, cross-engine reusable stateRuns as a standalone daemon process so 'KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines)'; MP mode serves one or more engine instances from a shared storeTiered offload across CPU memory, local disk (SSD) and remote backends, with reuse across requests, sessions and engine instancesPluggable storage and transport backends behind one interface: CPU RAM, local disk, Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, GDSPD disaggregation and KV transfer over NVLink, RDMA or TCP through transport layers such as NIXLNon-prefix KV reuse: reuse cached blocks at any prompt position, with selective recompute for quality recovery (CacheBlend)Production-grade KV observability: request- and token-level prefix-cache hit metrics, lifecycle and per-user usage metrics, Kubernetes health metricsPluggable SERDE interface for compression, token dropping and custom serialisation
Hardware caveats
Multi-platform support including a no-GPU development mode, but the fast paths are accelerator-specificThe KV-layout coupling is the sharp edge: LMCache's correctness depends on the engine's KV cache layout, and vLLM 0.26's fused/packed layout silently corrupted restores until it was fixedLayout changes are not version-gated - a vLLM PR changing the block layout broke existing LMCache versions with no error
Best for
Long-context agentic, multi-turn and RAG workloads where TTFT is dominated by repeated prefill of a shared prefixFleets serving the same corpus across multiple engines or replicas, where a shared KV store beats per-engine prefix cachesDeployments that want the engine decoupled from cache lifetime (daemon process, no fate-sharing)
Avoid for
Workloads with no prefix reuse and no KV pressure - the layer adds a hop for nothingPinning to an engine version without checking the KV layout, because the compatibility failure is silentCorrectness-critical serving without an output-equivalence test against a monolithic baseline - LMCache P/D has a documented 9/9 failure against exactly that oracle
notes
Research prototype or production system? Production infrastructure with heavy adoption (Apache-2.0, ~12k stars, active). It is a layer, not a serving engine: it plugs into vLLM, SGLang and others rather than replacing them. KV transfer path: two distinct paths that are often conflated. (1) PD handoff - 'Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL'. (2) Tiered offload - CPU RAM and local disk on the node, remote backends (Mooncake, InfiniStore, Redis/Valkey, S3, GDS) off-node. Correctness is the headline risk, and it is well documented upstream: LMCache #4463 records silent wrong KV on vLLM 0.26's packed layout ('A request served from the LMCache CPU tier produces different output than the same request computed from scratch. There is no exception and no warning'); Dynamo #14416 generalises it across three connectors and three LMCache issues; and vLLM #52617 shows the PD path itself diverging from a monolithic baseline at token 0 in 9/9 comparisons while native NIXL and Mooncake passed the same oracle. The pattern across all three is the same lesson: a KV cache is keyed on tokens but validated on layout, dtype, TP size, attention backend and adapter identity, and anything omitted from that key becomes silent corruption. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v0.5.5 published 2026-09-12, 344 contributors (distinct committers including anonymous), open_issues_count 751 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 11958 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.