CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Reusing a cached KV context across requests saves prefill compute but the KV tensors are large, so fetching them over the network dominates context-loading delay.

A CONTEXT-TRANSPORT codec, not a compute optimization: a custom tensor encoder exploits the distributional properties of KV caches (heavy-tailed channels, known positional structure) to encode the cache into a compact bitstream with negligible decode overhead, and the compression level is ADAPTED PER REGION of the context to current available bandwidth - raising compression when bandwidth drops, and recomputing a region on the fly when compression would cost too much quality.

Paper's own claim (abstract): compared to recent systems that reuse the KV cache, CacheGen REDUCES THE KV CACHE SIZE BY 3.5-4.3x AND THE TOTAL DELAY in fetching and processing contexts by 3.2-3.7x, 'with negligible impact on the LLM response quality'. COMPRESSION RATIO: 3.5-4.3x on the encoded KV stream (an entropy/structured codec, so the on-disk ratio is not a bit width). QUALITY COST: 'negligible impact' - no paired delta in the abstract. This is a network-bandwidth paper: the win requires a serving system that fetches KV from remote or shared storage.

SLUG RENAMED 2026-10-04 (parent agent): was 'cachegen', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2310.07240 verified via /abs/; v6. Venue SIGCOMM 2024 from Semantic Scholar and the arXiv comment field ('SIGCOMM'24'); SIGCOMM is NOT in this schema's venue enum, so venue='other' and the real venue is named in notes. Year = venue year 2024. Repo already has source qlab-cachegen-paper and a quantization/cachegen record. arXiv ID and title verified by fetching https://arxiv.org/abs/2310.07240 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2310.07240) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. Peer-reviewed venue confirmed as SIGCOMM 2024 (same). This is an additive track annotation; `venue` is unchanged at 'other' and no status change is implied by the track alone. VENUE RECLASSIFIED 2026-10-05 (schema authority pass, agent w5-schema): venue 'other' -> 'sigcomm', now that schemas/paper.schema.json carries a sigcomm member. The enum member was added only because this record existed and could be reclassified in the same commit - adding a venue nobody records is how this repo shipped 36 dead enum values. The venue was verified against the PUBLISHER'S OWN Crossref deposit, not an arXiv comment string, because this repo has already been burned by comment-derived venues. VENUE NOW VERIFIED FROM THE PUBLISHER'S DEPOSIT rather than from the arXiv comment and Semantic Scholar that the note above cites: Crossref for DOI 10.1145/3651890.3672274 returns container-title 'Proceedings of the ACM SIGCOMM 2024 Conference', event "ACM SIGCOMM '24", pages 38-56, issued 2024-08-04, type proceedings-article, parent volume 10.1145/3651890, with the title 'CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving' matching this record exactly. Resolved by BIBLIOGRAPHIC QUERY rather than by DOI guess, which is the discipline the repo's own crossref source record warns about after a ColBERT DOI turned out to point at a different paper.