Linear Transformers Are Secretly Fast Weight Programmers

Linearised softmax attention updates its state by a purely additive outer product k v^T, and the resulting fixed-size state has a memory capacity that cannot support precise recall.

Shows formally that linearised attention is equivalent to a fast weight programmer: a slow net learning by gradient descent to program the fast weights of another net through sequences of additive outer-product instructions. From that framing it derives the memory capacity limitation of additive-only linear attention and REPLACES the additive update with a delta-rule-like programming instruction, S_t = S_{t-1} + beta_t (v_t - S_{t-1}^T k_t) k_t^T, i.e. write the new value and correct the prediction by a learned step size beta_t rather than only accumulating. That correction step is what raises recall. The state is fixed-size and constant in sequence length, so no KV cache. This delta-rule state update is the direct ancestor of Gated DeltaNet, Kimi Delta Attention, and Qwen3-Next, and it is why 'delta rule' language appears throughout modern linear-attention work.

No throughput number. The claim is a state size versus recall Pareto improvement from the delta rule, evaluated as associative-recall and language-modelling perplexity at fixed state size; the theoretical contribution is the fast-weight-programmer equivalence and the derived capacity limit.

ICLR 2021 per the arXiv comment field; no journal-ref on the abs page. The paper is three years older than the wave it enabled: its delta rule is the mechanism now shipped in production inside Gated DeltaNet, Kimi Linear and Qwen3-Next, so 'in-upstream-engine' refers to the mechanism having shipped in shipped models, not to this 2021 code. Worth recording because it is the root citation for the delta rule and most downstream papers cite Gated DeltaNet instead. VENUE CORRECTED 2026-10-04 (paper-verification pass): venue/year iclr 2021 -> icml 2021. EVIDENCE: Title found in PMLR v139 (ICML 2021) at score 1.0 (Schlag, Irie, Schmidhuber); absent from the ICLR 2021 accepted-paper index. arXiv 2102.11174 carries NO comment field and NO journal_ref, so the record's note "ICLR 2021 per the arXiv comment field" cites a field that does not exist. Its delta-rule mechanism is shipped in production (Gated DeltaNet, Kimi Linear, Qwen3-Next), so the mechanism claim is unaffected; only the venue attribution was wrong. This is the clearest case in the batch of a venue taken from a field that does not exist on the source page. Both values retained here deliberately; status set to contested because a peer-reviewed attribution changed. arxiv_id unchanged (2102.11174). NOTE-VS-FIELD CONTRADICTION RESOLVED 2026-10-04 (paper-fields pass, agent w3p): no field changed, but the record's note and its `venue` field disagreed and the FIELD was right. The note said 'ICLR 2021 per the arXiv comment field', while venue='icml' 2021. Checked against the static PMLR v139 table of contents on 2026-10-04: it lists 'Linear Transformers Are Secretly Fast Weight Programmers' by Imanol Schlag, Kazuki Irie, Jurgen Schmidhuber, 'Proceedings of the 38th International Conference on Machine Learning', PMLR 139:9355-9366. v139 is confirmed as ICML 2021 by reading the volume landing-page header ('Volume 139: International Conference on Machine Learning, 18-24 July 2021'). So the paper IS at ICML 2021 and the note's ICLR claim came from the arXiv comment field, which is wrong. Note the trap this avoids: PMLR volume numbers are NOT venue-ordered, and v130 is the 35th ICLR-adjacent AISTATS volume, so an earlier title-index sweep in this pass initially matched this title against the WRONG volume; the volume->venue mapping was re-derived from each volume's landing-page header before drawing any conclusion. 2026-10-10: PMLR v139 ICML 2021 proceedings (idx1.py: https://proceedings.mlr.press/v139/) confirms "Linear Transformers Are Secretly Fast Weight Programmers" (Schlag, Irie, Schmidhuber, 38th ICML). Source pap-arch-2102-11174 notes cite ICLR 2021 per arXiv comment vs sco-pmlr-icml-2021-v139 which confirms ICML 2021. Sources disagree; status remains contested.