RWKV: Reinventing RNNs for the Transformer Era

Transformers parallelise during training but are quadratic and memory-hungry at inference; RNNs are linear and constant-memory at inference but do not parallelise during training. Architecture families are usually a choice, not both.

RWKV is formulated so the SAME model is a Transformer during training and an RNN during inference: a linear-attention formulation with time-mixing (WKV) and channel-mixing blocks, plus a receptance-gated formulation that lets the recurrence be expressed in a numerically stable closed form with no time-step hyperparameter. Constant memory and constant compute per token at inference.

Paper's own claims (abstract): scaled to 14B parameters, 'by far the largest dense RNN ever trained', performing ON PAR with similarly sized Transformers. That is a quality claim, not a speedup: the abstract states constant computational and memory complexity during inference without attaching a multiplier or a baseline engine.

arXiv ID 2305.13048 verified via abs-page meta tags on 2026-10-03; title matches. Venue verified via OpenReview with an explicit venueid, which is what makes this reliable: venue='EMNLP 2023 Findings', venueid='EMNLP/2023/Conference'. The important qualifier, which the dead agent's source note missed entirely: this is the FINDINGS track, not the main track - it is a real publication but a shorter one. Note also two ACL Anthology archive copies exist with venue=None (OpenReview.net/Archive), which is where the confusion with 'no venue' comes from; the Findings entry is the citable one. adoption=in-upstream-engine verified in source: transformers ships a models/rwkv/ directory. That is narrower than Mamba's support (no vLLM or SGLang model file, no dedicated backend), so the honest reading is that RWKV is servable in the reference library but has no first-class path in the major engines - recorded in-upstream-engine with that qualifier rather than in-production. open_weights=true: released RWKV models exist. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'findings' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2023.findings-emnlp.936, booktitle 'Findings of the Association for Computational Linguistics: EMNLP 2023'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/rwkv.json and data/papers/rwkv-reinventing-rnns.json were BOTH live records for arXiv 2305.13048 (RWKV: Reinventing RNNs for the Transformer Era), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. rwkv-reinventing-rnns.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2305.13048 with this title and the author list [Peng, Bo, Alcaide, Eric, Anthony, Quentin, Albalak, Alon], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "rwkv-reinventing-rnns". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (1083 chars vs 384) and is RETAINED here rather than discarded. Its text: Recurrent Weighted Key Value (RWKV) is formulated so the same model computes three ways: as a Transformer (parallel training), as an RNN (constant-cost sequential inference), and as a chunkwise-recurrent hybrid. The layer replaces softmax attention with a data-dependent exponential decay: for the time-mixing sublayer, WKV_t = (sum_i e^{-(t-1-i)w + k_i v_i^T) / (sum_i e^{-(t-1-i)w + k_i}) + D u_t, where w is a learned per-channel decay rate, so the attention weight at distance d is e^{-wd}, a fixed exponential window rather than a learned softmax. The channel-mixing sublayer is a standard position-wise feed-forward with squared ReLU. State is a fixed-size vector per channel of W per token, CONSTANT in sequence length, so there is no KV cache; RWKV was one of the first open architectures to make that concrete for LLMs, and its inference cost is genuinely independent of context length. Version lineage matters: RWKV-4 is the LSTM-style version above, RWKV-5 (Eagle) and RWKV-6 (Finch) change the mixing formulas, and RWKV-7 'Goose' moves to a delta-rule-style state update. speedup_reported: the retired record's text was materially fuller (499 chars vs 329) and is RETAINED here rather than discarded. Its text: The paper claims comparable or better performance to Transformers of similar size at 14B scale while being 10x faster at inference, with memory independent of sequence length. Hardware was A100/RTX-class GPUs. Important caveat that survived contact with practice: the '10x faster' figures come from the authors' own kernels and small-batch settings, and the sequential scan has the same fundamental character as Mamba's, so engine support and tuned throughput are much weaker than the paper implies. hardware_relevance UNION taken: this record had A100 40GB/80GB SXM4 and H100 SXM; the retired twin had A100 80GB SXM4, RTX 4090 and L40S. The twin's consumer-GPU entries reflect the paper's own reported A100/RTX-class testbed. AFFILIATIONS CORRECTED - AFFILIATION ERROR FOUND AND CORRECTED, and this is the widest gap in the sweep. arXiv HTML byline for 2305.13048v2 lists SIXTEEN distinct affiliations across the 33 authors - Generative AI Commons, EleutherAI, U. of Barcelona, Charm Therapeutics, Ohio State U., U. of C. Santa Barbara, Zendesk, Booz Allen Hamilton, Tsinghua University, Peking University, Storyteller.io, Crisis24, New York U., National U. of Singapore, Wroclaw U. of Science and Technology, and Databaker Technology. This record listed 'University of Washington', 'EleutherAI' and 'NVIDIA': University of Washington appears NOWHERE on the paper (Bo Peng is at Generative AI Commons / EleutherAI), NVIDIA appears NOWHERE, and fifteen of the sixteen real institutions were missing. The retired twin listed EleutherAI alone - an under-count rather than an error, which is the lesser failure but still wrong. The byline also carries an equal-first-authorship footnote covering Peng, Alcaide and Anthony with 'others listed alphabetically', so the byline order is not a contribution ranking and must not be read as one. VENUE AND THE CLAIMS THAT COMPETED FOR IT: EMNLP 2023 FINDINGS on both records - one of the few pairs where venue, track and year all agreed. The retired twin's evidence was the arXiv comment field; this record's is an explicit OpenReview venueid 'EMNLP/2023/Conference' with venue string 'EMNLP 2023 Findings', and the ACL Anthology DOI 10.18653/v1/2023.findings-emnlp.936 corroborates independently. The Findings qualifier is load-bearing and is preserved in venue_track: a Findings paper runs a shorter review path than the main track, so a filter of venue=emnlp with track=main must not return this paper. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.