Sparse Attention with Linear Units

Sparse attention is usually obtained by sparsifying SOFTMAX, which needs approximations of the softmax function and ties sparsity to the softmax's behaviour.

Replace the softmax with a ReLU: Rectified Linear Attention (ReLA). Sparsity emerges NATURALLY from the ReLU formulation - negative scores produce exactly zero output - rather than being imposed by a separate sparsity mechanism. Training stability is obtained with layer normalisation plus either a specialised initialisation or an additional gating function. A property no softmax-sparsified alternative has: heads can learn to attend to NOTHING for some queries, i.e. switch themselves off.

NOT A QUANTIFIED SPEEDUP. The abstract reports quality parity and structural efficiency: translation performance comparable to strong baselines across five machine translation tasks, with training and decoding speed SIMILAR to vanilla attention, plus high sparsity rate and head diversity and better source-target alignment accuracy than sparsified-softmax models. 'Similar speed to vanilla attention' plus emergent sparsity is the claim - the sparsity rate is a property, not a measured serving win.

arXiv ID 2104.07012 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. Venue verified against the ARCHIVE OF RECORD rather than the comment field: the arXiv comment says 'EMNLP2021', and the paper appears in the ACL Anthology volume 2021.emnlp-main at https://aclanthology.org/2021.emnlp-main.523/ whose title reads 'Sparse Attention with Linear Units - ACL Anthology' (located by scanning the volume listing; the neighbouring id 2021.emnlp-main.522 is a different paper, 'Editing Factual Knowledge in Language Models'). venue=emnlp, year=2021. adoption=research-only verified by absence: no ReLA or relu-attention path in vLLM's tree, SGLang's tree, or transformers' attention modules. Two cautions recorded because the name is a trap: (1) an OpenReview search for this title returns a DIFFERENT paper's records, 'SEA: Sparse Linear Attention with Estimated Attention Mask' (ICLR 2024 poster) - unrelated; (2) this 2021 encoder-decoder machine-translation work is not the ancestor of the linear-attention serving kernels in this repo - those descend from linear-transformers and gated/SSD variants (mamba-2, kimi-linear), which use softmax-derived kernel feature maps, not ReLU. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'main' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2021.emnlp-main.523, booktitle 'Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which.