LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

KV cache grows far past GPU memory in real deployments, and moving it to CPU/DRAM/SSD with existing ad-hoc paths is slow, so cache reuse across queries and across engines is impractical.

A dedicated KV cache layer between the engine and storage with an explicit connector interface, non-blocking save/load paths that overlap transfer with compute, and a store-back policy that decides what is worth persisting. The design goal is reuse ACROSS queries and ACROSS engines, not just offload.

The paper's own motivation is real-world usage statistics showing total stored KV cache growing far beyond GPU capacity; it claims to be 'the first and so far the most efficient open-source' KV cache layer. NO specific speedup multiple is stated in the abstract, so no multiple is recorded here rather than inventing one. Any TTFT reduction figure must be read from the paper's evaluation section together with its cache hit-rate assumption — on a cold workload with zero hits there is no gain by construction.

SLUG RENAMED 2026-10-04 (parent agent): was 'lmcache', colliding with the engines record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs (['affects']) all mean the engines record, so none needed repointing; papers are cited via `sources`, never by bare slug. Venue verified on OpenReview ('ICLR 2025 Poster'); arXiv comment empty. Adoption verified against two independent artefacts: the repo is actively pushed (2026-10-03), and vLLM ships LMCacheConnectorV1 and LMCacheMPConnector as documented connectors, including for disaggregated prefill. Being an in-upstream-engine CONNECTOR is different from being a vLLM subsystem — vLLM's own KV management is not LMCache. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. VENUE UNVERIFIED - FLAGGED CONTESTED 2026-10-04 (inference-gap pass, agent w3-papers-26). THIS RECORD SAYS venue=iclr AND I COULD NOT CONFIRM IT; the value is LEFT IN PLACE rather than rewritten, per the repo rule that a peer-reviewed attribution is reported and contested, never silently changed. WHAT WAS CHECKED: the OpenReview note-search index was queried for the exact title 'LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference'. It returns exactly ONE record, forum M1BI86jGg6, with content.venue 'CoRR 2025' and content.venueid 'dblp.org/journals/CORR/2025' - i.e. the DBLP mirror of the arXiv deposit itself, NOT a conference decision record. Compare the shape of a real ICLR acceptance in the same index, from this same pass: Fiddler returns 'ICLR 2025 Poster' with venueid 'ICLR.cc/2025/Conference' (forum N5fVv6PZGz), and Preble returns 'ICLR 2025 Poster' with venueid 'ICLR.cc/2025/Conference'. LMCache has no such record and no ICLR.cc venueid anywhere. Corpus checks are also negative: LMCache is absent from the ICLR 2025 accepted-title index, from PMLR v267 (ICML 2025), from the MLSys 2025 and 2026 indexes (61 and 135 papers), and from the NeurIPS 2024 index (4493 papers). arXiv 2510.09665's comment field carries no venue claim either. WHY THIS MATTERS AND WHY IT IS NOT SIMPLY OVERWRITTEN: LMCache is one of the most widely cited KV-cache systems of its generation and it IS a plausible ICLR 2025 paper in memory - which is exactly the conditions under which this repo has shipped wrong attributions twice before. The honest current state is 'claimed iclr, unverified', which is what 'contested' records. A future pass that finds an ICLR.cc venueid or an ICLR 2025 index entry for arXiv 2510.09665 should promote this back to verified; a future pass that finds none should correct venue to arxiv-preprint with venue_track unknown, and record THAT correction rather than leaving the field as an unverified assertion. status changed verified -> contested; confidence 0.84 -> 0.6; venue and year UNCHANGED (iclr, 2025); arxiv_id UNCHANGED (2510.09665). BOTH VALUES RETAINED. VENUE CORRECTED 2026-10-05 (wave 6, agent w6-papers): venue 'iclr' -> 'arxiv-preprint', venue_track 'main' -> 'unknown', status contested -> verified. The iclr value is GONE, not merely contested, and the reason is a DATE rather than a failed search. THE DECISIVE ARGUMENT. The arXiv API returns published 2025-10-08 for 2510.09665 (v2 2025-12-05). ICLR 2025 author notifications were January 2025 and its conference April 2025; the next ICLR cycle open after this preprint appeared is ICLR 2026. A paper first posted to arXiv in OCTOBER 2025 therefore CANNOT have been an ICLR 2025 paper - the decision preceded the preprint by roughly nine months. This is stronger than a negative index search because it does not depend on any index being complete: it is a consistency argument between two independently documented dates. Source w6p-arxiv-api-lmcache-timing. THE SEARCH REPRODUCES TOO, on a different endpoint. OpenReview api2 for 'LMCache' (source w6p-openreview-api2-lmcache-only-corr) returns exactly one titled, venued record for this paper: forum M1BI86jGg6, venue 'CoRR 2025', venueid 'dblp.org/journals/CORR/2025' - the arXiv deposit reflected back, since a dblp.org/journals/CORR venueid is the class that means 'no proceedings entry'. The other titled hits in the result set are a DIFFERENT paper ('Do Large Language Models Need a Content Delivery Network?', CoRR 2024, same Cheng/Du/Yao/Jiang group), a survey that CITES LMCache, and a 2027 submission. No ICLR.cc venueid appears anywhere. This independently confirms wave 4's finding rather than contradicting it. HOW THE ERROR GOT IN, since 'the search said so' is not good enough. This record's OWN source record pap-sys-lmcache-openreview states 'content.venue = ICLR 2025 Poster' and its url is not a forum page but an api2 SEARCH query - a claim of an exact venue string with no forum id behind it, against a search that returns something different today. pap-sys-lmcache-arxiv repeats the same sentence. A search URL is not a citation: it cannot be re-read to confirm what it said at the time it was read, which is exactly why this claim was able to sit in the repo marked 'verified' and then survive a wave that could not reproduce it. Both source notes are left in place UNCHANGED on the record that cites them - they are the provenance of the error, and rewriting them would destroy the evidence for how it happened. What is corrected is the paper record. WHAT IS NOT CHANGED: adoption=in-upstream-engine stands on independent grounds and is unaffected by the venue. vLLM ships LMCacheConnectorV1 and LMCacheMPConnector as documented connectors, including for disaggregated prefill, and the LMCache repository is actively pushed. Being an in-upstream-engine CONNECTOR is different from being a vLLM subsystem - vLLM's own KV management is not LMCache. arxiv_id unchanged (2510.09665). THE ABSTRACT FIGURE THIS RECORD NOW OWES ITS READERS, since the venue correction raises rather than lowers confidence and the headline number deserves to be stated: LMCACHE reports UP TO 15x throughput improvement when combined with vLLM, on multi-round question answering and document analysis. That is a with-LMCache-against-without-LMCache comparison on a prefix-reuse-heavy workload, so it is bounded by the cache hit rate by construction - on a cold single-turn workload with zero prefix hits there is no gain at all. The paper's own operational finding is the more useful one for a deployment reader: context truncation, widely applied in industry, roughly HALVES the prefix cache hit ratio. That is a real operational consequence of a widely-applied optimisation and is worth more than the 15x.