MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

PREFILL for long-context LLMs is quadratic in context length, and a single sparse-attention pattern cannot serve all long-context tasks because attention structure differs by head type and by task.

Dynamic sparse attention selected per-head at prefill time. Heads are classified into retrieval heads and non-retrieval heads; each class gets a different prefill pattern - vertical-slash patterns for retrieval heads (which need sparse content-based access) and block-sparse patterns for others, with per-head pattern and budget chosen online from the input. Then optimised GPU kernels execute the sparse patterns.

Paper's own claim (abstract): reduces prefill inference latency by UP TO 10x ON AN A100 while maintaining accuracy, evaluated on InfiniteBench, RULER, PG-19 and needle-in-a-haystack with LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K and Qwen2-128K. Hardware named (A100), and the baseline class is dense attention prefill - this is the best-anchored prefill claim in the batch. Note it is a PREFILL result: it says nothing about decode tok/s.

arXiv ID 2407.02490 verified via abs-page meta tags on 2026-10-03; title matches. Venue confirmed by two sources: arXiv comment 'Accepted at NeurIPS 2024 (Spotlight)' and OpenReview venue='NeurIPS 2024 spotlight', venueid='NeurIPS.cc/2024/Conference'. adoption=research-only verified by absence, and this one required care: an early GitHub code search suggested 'minference repo:vllm-project/vllm' returned hits, but the only matches were docs/deployment/integrations/kserve.md and llm-d.md - incidental mentions in deployment docs, not an implementation. A full recursive file listing of vllm-project/vllm@main contains NO path with 'minference' in it (checked via the git tree API). There is also no minference in SGLang's models or attention directories and no MInference cache class in transformers. The vLLM ecosystem's *idea* of this work (per-head sparse prefill with configurable patterns) is the general sparse_attention_config / hybrid_linear_attn_backend machinery, but MInference itself is not shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.