Kimi Linear: An Expressive, Efficient Attention Architecture

Linear attention has been adopted mainly for efficiency at the cost of quality, and the claim that it can beat full attention under FAIR comparisons had not been established for short-context, long-context and RL-scaling regimes simultaneously.

Kimi Delta Attention (KDA), a linear attention module that EXTENDS Gated DeltaNet with a finer-grained gating mechanism so a limited finite-state RNN memory is used more effectively. The chunkwise algorithm uses a specialised variant of Diagonal-Plus-Low-Rank transition matrices, which cuts computation substantially versus the general DPLR formulation while staying closer to the classical delta rule. Architecture: layerwise HYBRID of KDA and Multi-Head Latent Attention (MLA).

Paper's own claims (abstract): reduces KV CACHE USAGE by up to 75% and achieves up to 6x DECODING THROUGHPUT at 1M context, while outperforming full MLA with a sizeable margin under an identical training recipe across all evaluated tasks. The 6x is a decode-throughput-at-1M-context figure - the regime this architecture targets - and the 75% is a cache-size reduction, not a bandwidth claim.

arXiv ID 2510.26692 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. venue=arxiv-preprint: the arXiv comment says only 'Kimi Linear tech report' and no OpenReview record for this title exists (the search surfaces Kimi-VL Technical Report and unrelated KDA/linear-attention papers, not this one). adoption=in-upstream-engine verified in source, twice over: SGLang ships python/sglang/srt/models/kimi_linear.py, and vLLM has csrc/flashkda_registration.cpp (the KDA kernel) plus a GatedDeltaNet attention backend at vllm/v1/attention/backends/gdn_attn.py. So both major engines carry KDA support - among the newest methods in this batch, one of the best-integrated. open_weights=true: the paper states it releases pre-trained and instruction-tuned checkpoints plus the KDA kernel and vLLM implementations. Distinct from moba (sparse attention) and from deepseek-v2-mla (MLA) even though all three involve Kimi or MLA - the hybrid here is KDA + MLA. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/kimi-linear.json and data/papers/kimi-linear-kda.json were BOTH live records for arXiv 2510.26692 (Kimi Linear: An Expressive, Efficient Attention Architecture), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. kimi-linear-kda.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2510.26692 with this title and the author list [Kimi Team, Zhang, Yu, Lin, Zongyu, Yao, Xingcheng], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "kimi-linear-kda". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (926 chars vs 480) and is RETAINED here rather than discarded. Its text: Introduces Kimi Delta Attention (KDA), a linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism so the model makes better use of a limited finite-state RNN memory, plus a bespoke chunkwise algorithm built on a specialised Diagonal-Plus-Low-Rank (DPLR) transition matrix that substantially reduces computation versus the general DPLR formulation. The shipped model, Kimi Linear 48B-A3B, is an MoE hybrid: 3B parameters active per token out of 48B total (about 6.25 percent active), with a gated-delta linear-attention pattern rather than Mamba's scalar-selective SSM. The attention pattern is hybrid: a majority of layers use KDA and a minority use full attention, which is the specific design choice worth carrying into serving work because the full-attention layers are what bound the KV cache. Constant-size state within the linear layers means those layers add no length-dependent memory. speedup_reported: the retired record's text was materially fuller (551 chars vs 392) and is RETAINED here rather than discarded. Its text: Claims to be the first hybrid linear-attention architecture to outperform full attention under fair comparison across short-context, long-context and RL-scaling regimes, and reports lower KV cache usage and faster decoding than full-attention baselines of comparable quality. The report gives cache-size and decode-throughput comparisons rather than a single speedup ratio; hardware is reported as H200/H800-class. Treat the 'outperforms full attention' claim as the authors' own fair-comparison framing rather than an independently reproduced result. AFFILIATIONS: left as-is on this record; the sweep verified it against the paper. VENUE AND THE CLAIMS THAT COMPETED FOR IT: No peer-reviewed venue, agreed by both records, and the negative is specific rather than generic: the arXiv comment reads only 'Kimi Linear tech report', and an OpenReview search for this title returns neither an acceptance nor even an arXiv deposit - the nearest hits are Kimi-VL Technical Report, Kimi k1.5, a Kimi-Dev ICLR 2026 poster, and a batch of ICLR 2027 linear-attention submissions (Complex KDA, DASC, FReS, LeapQuant, SDG) that are OTHER PAPERS with 'KDA' in the title. So there is no venue claim to copy and none to find, and arxiv-preprint is the honest value rather than a placeholder. The retired twin's `mechanism` was materially fuller and is retained: it names the chunkwise DPLR specialisation and the shipped 48B-A3B configuration, and its hardware list added AMD MI300X, which is preserved in the union. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.