RWKV: Reinventing RNNs for the Transformer Era
paper
Fields
id
sources/arch-rwkvPublisher
Peng et al. / EleutherAI (arXiv)
Kind
paper
Published
2023-05-22
Accessed
2026-10-03
notes
RWKV is NOT an SSM. It is an RNN whose time-mixing is derived from a linear-attention formulation (WKV / receptance-weighted key-value), so it has a constant-size recurrent state but is not a structured state-space model: no diagonal state matrix, no convolutional scan. Uses a token-shift mixing of the current and previous inputs rather than a learned state-space discretization. Cite this whenever RWKV is grouped with Mamba; the distinction matters for the cost model.
Notes
RWKV is NOT an SSM. It is an RNN whose time-mixing is derived from a linear-attention formulation (WKV / receptance-weighted key-value), so it has a constant-size recurrent state but is not a structured state-space model: no diagonal state matrix, no convolutional scan. Uses a token-shift mixing of the current and previous inputs rather than a learned state-space discretization. Cite this whenever RWKV is grouped with Mamba; the distinction matters for the cost model.
Cross-references
This record references no other records.