Efficient Memory Management for Large Language Model Serving with PagedAttention

PagedAttention/vLLM paper (arXiv 2309.06180, SOSP 2023). PagedAttention: KV cache managed in fixed-size blocks like OS virtual memory. Near-zero waste in KV cache memory, flexible sharing within and across requests. vLLM improves throughput 2-4x vs FasterTransformer and Orca at same latency. Improvement more pronounced with longer sequences, larger models, more complex decoding.