InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (Liu et al.)

Speculative KV prefetch compatible with offloading-based serving; not a quantization scheme despite the title word 'quantization' appearing nowhere in its mechanism.