FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

Attention engine using block-sparse and composable KV-cache formats to address storage heterogeneity, JIT-compiled attention templates, and a load-balanced scheduler that stays compatible with CUDAGraph's static configuration. Reports 29-69% inter-token-latency reduction vs compiler backends, 28-30% for long context, 13-17% with parallel generation.