Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Mamba and attention looked like different families of sequence models, and there was no principled account of when or why one should be preferred - which made it hard to tell whether the SSM's weaknesses were architectural or incidental.

State Space Duality (SSD). Show that SSMs and variants of attention are connected through decompositions of structured semiseparable matrices, which yields a single framework in which Mamba's selective SSM can be re-expressed and then refined. The refined Mamba-2 layer uses a structured scalar-descent formulation of the delta rule over an SSD representation, which maps onto matrix-multiplication shapes that tensor cores handle natively - the reason the speedup exists.

Paper's own claim (abstract): the Mamba-2 core layer, a refinement of Mamba's selective SSM, is 2-8x FASTER than Mamba's layer while remaining competitive with Transformers on language modelling. The baseline is Mamba's own layer - this is a layer-level kernel claim, not an end-to-end serving claim, and should be recorded as such.

arXiv ID 2405.21060 verified via abs-page meta tags on 2026-10-03; title matches. Venue icml confirmed twice: arXiv comment 'ICML 2024' and OpenReview DBLP mirror with venue='ICML 2024'; a direct OpenReview search for the title returns a DBLP record rather than an ICML.cc venueid, so the comment field plus DBLP is the evidence chain. adoption=in-upstream-engine verified in source: vLLM has vllm/model_executor/models/mamba2.py AND a dedicated v1 attention backend vllm/v1/attention/backends/mamba2_attn.py, plus mamba/ops/replayssm_config.py; SGLang ships mamba2.py; transformers ships models/mamba2. A dedicated engine attention backend is stronger evidence than a model file alone, so the speedup claim is reflected in shipped scheduling code. Note the FA2 comparison in the mechanism: Mamba-2's 2-8x is against the Mamba layer, NOT against FlashAttention-2 - easy to misread and worth keeping explicit. open_weights=true. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/mamba-2.json and data/papers/mamba-2-ssd.json were BOTH live records for arXiv 2405.21060 (Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. mamba-2-ssd.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2405.21060 with this title and the author list [Tri Dao, Albert Gu], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "mamba-2-ssd". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (1027 chars vs 472) and is RETAINED here rather than discarded. Its text: Establishes state space duality (SSD): a large class of structured matrices are semiseparable and can be decomposed either as an SSM recurrence or as an attention-like sum, so SSMs and attention variants are two views of the same object rather than competing families. Exploiting this yields Mamba-2, whose core layer is a refinement of the Mamba selective SSM in two ways that matter for hardware. (1) It moves to a much larger state, d_state=128 instead of 16, grouped into heads, which is what recovers recall. (2) It replaces the scalar-per-head recurrence with a matrix recurrence expressible as a single SSD algorithm: within a chunk compute the attention-like form, across chunks apply the semiseparable decay matrix. This turns the scan into block-parallel matmuls that DO map onto tensor cores, so the sequential-scan bottleneck is largely removed. State remains fixed-size per layer (batch, n_heads, head_dim, d_state), so as with Mamba-1 there is still no KV cache and still no length-dependent decode memory growth. speedup_reported: the two records carried DIFFERENT text for the same paper. This record's text is retained and the retired record's variant is quoted so the difference is visible: "The Mamba-2 core layer is 2-8x faster than the Mamba-1 selective SSM while remaining a drop-in replacement and remaining competitive on upstream tasks. Measured on A100 GPUs against the authors' own Mamba-1 reference implementation. This is a kernel-level statement about one layer, not an end-to-end model throughput number, and the 2-8x range is wide because it varies with sequence length and batch size." AFFILIATIONS: left as-is on this record; the sweep verified it against the paper. VENUE AND THE CLAIMS THAT COMPETED FOR IT: ICML 2024 on both records - a clean duplicate with no venue disagreement. Evidence: arXiv comment 'ICML 2024' plus the OpenReview DBLP mirror 'ICML 2024' / dblp.org/conf/ICML/2024. Worth recording that a direct OpenReview search for this title returns the DBLP record rather than an ICML.cc venueid, so unlike most ICML papers in this corpus the venueid leg is absent and the comment field is doing real work. Two substantive prose differences between the records are preserved rather than resolved, because both are defensible: the retired twin's `mechanism` is materially fuller and describes the SSD framework as the reason SSM training became hardware-coherent, which is the more useful gloss; and its `speedup_reported` adds that the 2-8x is measured on A100 against the authors' own Mamba-1 reference implementation, which this record states less explicitly. The '2-8x is against the Mamba LAYER, not against FlashAttention-2' warning from this record is retained in full. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.