Retentive Network: A Successor to Transformer for Large Language Models

Transformers need attention to model token interactions; recurrent models cannot parallelise training. The choice of sequence-mixing primitive forces that trade-off, and no prior alternative offered training parallelism, O(1) inference and Transformer-level quality at once.

Retention: a sequence-mixing primitive derived by a theoretical link between recurrence and attention, expressed in THREE equivalent paradigms - parallel (for training), recurrent (for O(1) inference), and chunkwise recurrent (for long sequences: encode each chunk in parallel while recurrently summarising the chunks). All three are the same function, so one set of weights serves training and serving without conversion.

The abstract makes NO numeric speedup claim: it reports 'favourable scaling results, parallel training, low-cost deployment, and efficient inference'. The O(1) inference and improved decoding throughput/latency/GPU-memory claims are structural consequences of the recurrent form, stated without a multiplier or a baseline engine. Anything quantified here would be invented.

arXiv ID 2307.08621 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. VENUE: recorded as arxiv-preprint on EVIDENCE, which differs from the common citation. RetNet is very often cited as 'RetNet, ICLR 2024' but OpenReview forum UU9Icwbhin returns venueid='ICLR.cc/2024/Conference/Rejected_Submission' - rejected at ICLR 2024; a second OpenReview record sxZlp9ZoHD for the same authors and title returns venueid='NeurIPS.cc/2024/Conference/Rejected_Submission' - also rejected at NeurIPS 2024. No accepted venue found on OpenReview or via Semantic Scholar (429 unavailable). Per the rules, unverified venue becomes arxiv-preprint with this note rather than an assumed ICLR. adoption=research-only verified by absence, which is a real check: neither vLLM's nor SGLang's model directory contains a retnet module (vLLM registry has no RetNet entry; SGLang's 248-file models directory has no retnet/rwkv entry), and the only live repo found is the community 'syncdoth/RetNet' (228 stars, last pushed 2024-03-12) rather than an official implementation. So the three-paradigm idea is genuinely research-only. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/retnet.json and data/papers/retnet-retentive-network.json were BOTH live records for arXiv 2307.08621 (Retentive Network: A Successor to Transformer for Large Language Models), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. retnet-retentive-network.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2307.08621 with this title and the author list [Sun, Yutao, Dong, Li, Huang, Shaohan, Ma, Shuming], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "retnet-retentive-network". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (968 chars vs 422) and is RETAINED here rather than discarded. Its text: Proposes retention, a sequence model with three equivalent computation paradigms: parallel (for training), recurrent (for O(1)-memory inference) and chunkwise recurrent (a numerically stable training form). The recurrence is a fixed-decay exponential: S_n = gamma * S_{n-1} + K_n^T V_n with gamma = alpha^{1/L} a learned per-head decay, so the implicit attention weight between positions i and j is alpha^{(i-j)/L}, a deterministic geometric decay with no softmax and no data-dependent denominator. Three retention modes are used in parallel with different decay rates (a 1x1, a 1/4, and a 1/16 rate), the fastest decay on the highest-resolution stream, plus a GroupNorm-style normalisation, and Grouped-Retention or MSR (multi-scale retention) with a GroupNorm-based normalisation replacing softmax denominator. State is a fixed-size (d_model, d_head) matrix per head, CONSTANT in sequence length, so there is no KV cache and decode memory does not grow with context. speedup_reported: the retired record's text was materially fuller (518 chars vs 373) and is RETAINED here rather than discarded. Its text: Claims 7x faster training, 5x faster inference and 4x lower inference memory versus a standard Transformer of equivalent modelling power, plus parallelizable scaling to 128K context. Hardware was A100-class. These are the authors' own numbers on their own implementation and scale-law comparison; the 7x/5x/4x triple did not reproduce in production. RetNet did not become a shipped architecture in any mainstream engine: no retnet entry in llama.cpp's arch registry and no native row in the vLLM supported-models list. hardware_relevance was EMPTY here while the retired twin listed A100 80GB SXM4 and H100 SXM; populated from the twin. open_weights: open_weights left null: the retired twin said true, but RetNet shipped NO model weights - its own notes record that only aka.ms/retnet was promised - so true was wrong. AFFILIATIONS CORRECTED - arXiv HTML byline for 2307.08621v4 gives two affiliations: 'Microsoft Research' and 'Tsinghua University'. This record said Microsoft Research, the retired twin said 'Microsoft Research Asia'; the paper's own marker is the undifferentiated 'Microsoft Research', so that is what the field carries, with Tsinghua added - Jianyong Wang and Furu Wei are Tsinghua, and the paper's project link aka.ms/GeneralAI is Tsinghua's general-AI lab. VENUE AND THE CLAIMS THAT COMPETED FOR IT: NO peer-reviewed venue, and the ICLR story is settled - RetNet is widely cited as 'ICLR 2024' and it was REJECTED there. The OpenReview search for this exact title with the Sun/Dong/Huang/Ma author overlap returns exactly one corroborated record: venue 'Submitted to ICLR 2024', venueid 'ICLR.cc/2024/Conference/Rejected_Submission', forum UU9Icwbhin. There is no acceptance forum and no proceedings entry, so venue=arxiv-preprint is correct on evidence rather than by default - and it is the SAME shape as the Mamba bait, which is why both are recorded together. Note the contrast with S4, also an ICLR-family SSM paper, which WAS accepted at ICLR 2022: 'an ICLR SSM paper' is not a venue. The two records did not disagree on venue here, which is itself the point: the duplicate created an audit burden without even producing a contradiction to resolve. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.