Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv:2510.26692)

Fields

id

sources/model-cn-kimi-linear-paper

URL

https://arxiv.org/abs/2510.26692

Publisher

arXiv / Moonshot AI

Kind

whitepaper

Published

2025-10-30

Accessed

2026-10-03

notes

Primary source for the Kimi Delta Attention (KDA) hybrid pattern: full attention every 4th layer, KDA on the rest. config.json linear_attn_config enumerates the layers explicitly - full_attn_layers [4,8,12,16,20,24,27] (7 layers) and 20 kda_layers for Kimi-Linear-48B-A3B. This is the source for counting only the full-attention layers toward KV cache.

Notes

Primary source for the Kimi Delta Attention (KDA) hybrid pattern: full attention every 4th layer, KDA on the rest. config.json linear_attn_config enumerates the layers explicitly - full_attn_layers [4,8,12,16,20,24,27] (7 layers) and 20 kda_layers for Kimi-Linear-48B-A3B. This is the source for counting only the full-attention layers toward KV cache.

Cross-references

This record references no other records.