InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

In offloading-based serving, fetching the KV cache from host memory stalls the compute core; fetching all of it wastes bandwidth that only a few entries would have used.

Speculative KV PREFETCH: use the current layer's inputs plus PART of the next layer's query weight and key cache to do a MINIMAL REHEARSAL that predicts which few tokens the next attention layer will need, then prefetch only those KV entries from host memory. The prediction is deliberately approximate and cheap; the point is that the compute core stays busy instead of waiting on a bulk fetch.

Paper's own claim (abstract): improves the overall performance of a modern OFFLOADING-BASED system by UP TO 3.00x compared to prior KV cache management methods, while offering substantially better model accuracy. Baseline class is named (prior KV-management methods on an offloading system). COMPRESSION RATIO: none - this moves bytes, it does not shrink them. QUALITY COST: 'substantially better accuracy' is a relative claim; unlike KIVI/KVQuant this method never quantizes, so there is no quality-vs-bits tradeoff to report.

arXiv ID 2406.19707 verified via /abs/. Venue OSDI 2024 confirmed by the arXiv comment field ('OSDI 2024') and Semantic Scholar; year = venue year 2024. The gain is entirely conditional on running an offloading-based system with a slow host link - on a single-GPU high-bandwidth setup there is nothing to prefetch from. arXiv ID and title verified by fetching https://arxiv.org/abs/2406.19707 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2406.19707) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: osdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.