InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

Speculatively prefetches only the KV entries the current query will attend to, using an entry-prediction model run on CPU. It is the strongest available evidence that KV reads in long-context decode are a bandwidth wall rather than a math problem - the whole method is about issuing fewer, more targeted bytes - and it is why the decode-attention record lists speculative KV prefetch alongside fp8 KV and sparse/linear attention as the things that actually move this class.