inference/research
verifiedhigh · 0.92updated 2026-10-03

RWKV: Reinventing RNNs for the Transformer Era

paper

Fields

id
sources/arch-rwkv
Publisher
Peng et al. / EleutherAI (arXiv)
Kind
paper
Published
2023-05-22
Accessed
2026-10-03
notes
RWKV is NOT an SSM. It is an RNN whose time-mixing is derived from a linear-attention formulation (WKV / receptance-weighted key-value), so it has a constant-size recurrent state but is not a structured state-space model: no diagonal state matrix, no convolutional scan. Uses a token-shift mixing of the current and previous inputs rather than a learned state-space discretization. Cite this whenever RWKV is grouped with Mamba; the distinction matters for the cost model.

Notes

RWKV is NOT an SSM. It is an RNN whose time-mixing is derived from a linear-attention formulation (WKV / receptance-weighted key-value), so it has a constant-size recurrent state but is not a structured state-space model: no diagonal state matrix, no convolutional scan. Uses a token-shift mixing of the current and previous inputs rather than a learned state-space discretization. Cite this whenever RWKV is grouped with Mamba; the distinction matters for the cost model.

Cross-references

This record references no other records.