FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

FlashInfer paper (arXiv 2501.01005, MLSys 2025). Customizable attention engine for LLM serving. Block-sparse format and composable formats for KV-cache storage heterogeneity. JIT compilation for attention template customization. Load-balanced scheduling compatible with CUDAGraph. Integrated into SGLang, vLLM, MLC-Engine. 29-69% inter-token-latency reduction vs compiler backends, 28-30% latency reduction for long-context, 13-17% speedup for parallel generation.