Kimi Linear: An Expressive, Efficient Attention Architecture

Fields

id

sources/arch-kimi-linear

URL

https://arxiv.org/abs/2510.26692

Publisher

Moonshot AI (arXiv)

Kind

paper

Published

2025-10-30

Accessed

2026-10-03

notes

KDA (Kimi Delta Attention), a gated-delta-rule linear attention with a bespoke chunkwise algorithm using a specialised DPLR transition. Interleaves KDA with full attention in a uniform 3:1 ratio, reducing KV cache 'by up to 75%' and giving 'up to 6x decoding throughput' at 1M context; Figure 7b measures a 2.3x speedup at 1M context and the paper notes a 'maximum hybrid efficiency ratio of 3:1 compared to full attention' for I/O-bound decoding.

Notes

KDA (Kimi Delta Attention), a gated-delta-rule linear attention with a bespoke chunkwise algorithm using a specialised DPLR transition. Interleaves KDA with full attention in a uniform 3:1 ratio, reducing KV cache 'by up to 75%' and giving 'up to 6x decoding throughput' at 1M context; Figure 7b measures a 2.3x speedup at 1M context and the paper notes a 'maximum hybrid efficiency ratio of 3:1 compared to full attention' for I/O-bound decoding.

Cross-references

This record references no other records.