inference/research
draftlow · 0.50updated 2026-10-04

RetNet

recurrent
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
models/retnet-arch
Family
retnet
Vendor
Microsoft Research
Release year
2023
Architecture
recurrent
Context scaling method
native
Attention variant
retention (multi-scale exponential decay)
notes
RetNet is an architecture paper record (arXiv:2307.08621), not a released checkpoint: no config.json, no served weights, and no softmax KV cache. The KV-geometry fields are undefined for retention and left null. Retentive Network (RetNet) per arXiv:2307.08621. Three equivalent paradigms: parallel (training), recurrent (O(1) inference), and chunkwise recurrent (long sequences). The recurrent form gives O(1) per-token inference cost. 'Favourable scaling results, parallel training, low-cost deployment, and efficient inference' (no numeric speedup in abstract). Very often cited as ICLR 2024 but OpenReview forum returns venueid='ICLR.cc/2024/' suggesting rejection. Research-only adoption. [ID renamed from 'retnet-retentive-network' to 'retnet-arch' to avoid collision with papers/retnet-retentive-network. Source fixed from non-existent 'retnet' to 'pap-arch-2307-08621'.] CONTEXT_SCALING_METHOD (native): architecture record: multi-scale exponential-decay retention, no RoPE.

Notes

RetNet is an architecture paper record (arXiv:2307.08621), not a released checkpoint: no config.json, no served weights, and no softmax KV cache. The KV-geometry fields are undefined for retention and left null. Retentive Network (RetNet) per arXiv:2307.08621. Three equivalent paradigms: parallel (training), recurrent (O(1) inference), and chunkwise recurrent (long sequences). The recurrent form gives O(1) per-token inference cost. 'Favourable scaling results, parallel training, low-cost deployment, and efficient inference' (no numeric speedup in abstract). Very often cited as ICLR 2024 but OpenReview forum returns venueid='ICLR.cc/2024/' suggesting rejection. Research-only adoption. [ID renamed from 'retnet-retentive-network' to 'retnet-arch' to avoid collision with papers/retnet-retentive-network. Source fixed from non-existent 'retnet' to 'pap-arch-2307-08621'.]

CONTEXT_SCALING_METHOD (native): architecture record: multi-scale exponential-decay retention, no RoPE.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.