FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

Serving attention is not one problem: paged KV layouts, variable batch, head counts and context lengths each stress a different part of the memory hierarchy, so a single hardcoded attention kernel leaves large amounts of performance on the table across the workload distribution.

An attention ENGINE rather than a kernel. Three components: (1) block-sparse and composable KV formats so heterogeneous KV-cache layouts are expressed declaratively and memory access stays coalesced with less redundancy; (2) a customisable JIT-compiled attention template generated per configuration; (3) a load-balanced scheduling algorithm that adapts to request dynamism while remaining compatible with CUDAGraph's static-shape requirement.

Paper's own claims (abstract), against state-of-the-art LLM serving solutions and compiler backends: 29-69% inter-token-latency reduction on the LLM serving benchmark vs compiler backends; 28-30% latency reduction for long-context inference; 13-17% speedup for parallel generation. These are percentage latency reductions with named baseline class, not bare multipliers.

arXiv ID 2501.01005 verified via abs-page meta tags on 2026-10-03; title matches. Venue mlsys confirmed by TWO independent sources: the arXiv comment field reads 'Accepted by MLSys 2025' and OpenReview returns venue='MLSys 2025', venueid='MLSys.org/2025/Conference' for the exact title. The dead agent's source note said 'comment-field evidence, not independently confirmed via OpenReview' - now confirmed. adoption=in-upstream-engine verified in source, not inferred: SGLang ships python/sglang/srt/layers/attention/flashinfer_backend.py (HTTP 200), vLLM ships vllm/v1/attention/backends/flashinfer.py (120 KB) and triton_flashinfer.py, and the abstract itself names SGLang, vLLM and MLC-Engine as integrators. This is the most widely deployed attention engine after FlashAttention itself. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: mlsys has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.