MoBA: Mixture of Block Attention for Long-Context LLMs

Sparse attention for long context is usually a fixed pattern chosen ahead of time, which either misses genuinely distant information or wastes budget on tokens that do not matter.

Apply the principles of Mixture-of-Experts to attention: divide the context into blocks, let each query ROUTE to the top-k most relevant blocks via block-level scoring (mean-pooled keys, then a coarse-to-fine scoring), and attend only within those blocks. Because routing is per-query and input-dependent, the model LEARNS where to attend rather than having a bias imposed. Because it is block-level, the sparse access pattern is regular and hardware-friendly.

NOT A QUANTIFIED SPEEDUP. The abstract's substantive claim is DEPLOYMENT, not a multiple: 'MoBA has already been deployed to support Kimi's long-context requests'. It also claims superior long-context performance versus prior methods and the key practical property that it transitions SEAMLESSLY between full and sparse attention. The absence of a number is the fact: no speedup with a baseline appears in the abstract.

arXiv ID 2502.13189 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says the arXiv comment is only '15 pages' with no venue. OpenReview returns venue='NeurIPS 2025 spotlight', venueid='NeurIPS.cc/2025/Conference' for the exact title and the Lu/Jiang/Liu/Du author list. venue=neurips, year=2025. adoption=in-production - the strongest adoption claim in the whole attention batch and it comes from the paper's own abstract, not inference: MoBA is deployed to serve Kimi's long-context requests in production. Worth reconciling with the related kimi-linear record, which is a DIFFERENT approach (linear/hybrid attention) released by the same lab; MoBA is sparse-attention, Kimi Linear is linear attention, and both are Kimi long-context work. open_weights=false: the repo ships the method, not a released checkpoint of the served MoBA model. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.