Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Subquadratic alternatives to attention - linear attention, gated convolution, recurrent models, structured state space models - had not matched attention on language, and the paper diagnoses why: they cannot perform CONTENT-BASED reasoning, because their state transition and output are input-independent functions of the sequence.

Selective state space models. Two changes to the SSM: (1) make the SSM parameters - the discretisation step size and the state matrix - FUNCTIONS OF THE INPUT, so the model selectively propagates or forgets information per token (this is the 'selectivity' and it is what enables content-based reasoning, but it also breaks the fast convolutional form); (2) to recover speed, design a hardware-aware parallel algorithm for the recurrent mode that scans the sequence in a memory-efficient layout. Mamba is then just these selective SSM blocks with no attention and no MLP.

Paper's own claims (abstract): 5x higher inference throughput than Transformers, with linear scaling in sequence length; Mamba-3B outperforms same-size Transformers and matches Transformers twice its size on language modelling. The throughput claim's baseline class is 'Transformers' generally, not a named engine, so treat 5x as the paper's own generalisation rather than a reproducible decode-tok/s figure.

arXiv ID 2312.00752 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. VENUE CORRECTED - this is the most consequential venue fix in the batch. The dead agent's source note said 'The widely cited ICLR 2024 venue was not confirmed from the arXiv metadata, so venue is left as arxiv-preprint'. ICLR 2024 is NOT this paper's venue: OpenReview forum AL1fq05o7H (same two authors, same title) returns venueid='ICLR.cc/2024/Conference/Rejected_Submission'. The paper was REJECTED at ICLR 2024 and published at COLM 2024 - OpenReview forum tEYskw1VY2 returns venue='COLM', venueid='colmweb.org/COLM/2024/Conference'. venue=colm, year=2024. Anyone citing 'Mamba, ICLR 2024' is citing a rejection. adoption=in-upstream-engine verified in source: vLLM registry lists Mamba, Mamba2, Jamba and LFM2, with vllm/model_executor/models/mamba.py, mamba2.py, jamba.py and backends mamba1_attn.py / mamba2_attn.py / gdn_attn.py; SGLang ships mamba.py, mamba2.py and falcon_mamba.py; transformers ships models/mamba, mamba2, rwkv, falcon_mamba. Also note vLLM has a csrc/flashkda_registration.cpp, which is Kimi Delta Attention - the successor mechanism, recorded under kimi-linear. state-spaces/mamba is actively maintained (pushed 2026-07-22, 18.9k stars). open_weights=true. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: colm has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/mamba.json and data/papers/mamba-selective-ssm.json were BOTH live records for arXiv 2312.00752 (Mamba: Linear-Time Sequence Modeling with Selective State Spaces), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. mamba-selective-ssm.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2312.00752 with this title and the author list [Albert Gu, Tri Dao], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "mamba-selective-ssm". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (1199 chars vs 570) and is RETAINED here rather than discarded. Its text: Makes the SSM parameters functions of the input, which is the whole contribution. The layer is a selective SSM: input x is projected to (delta, B, C) where delta_t = softplus(W_delta x_t + bias) is data-dependent, B_t and C_t are data-dependent, and the state updates h_t = exp(delta_t * A) h_{t-1} + delta_t * B_t x_t^T followed by y_t = C_t h_t + D x_t. Because B, C and delta now depend on the token, the model can selectively propagate or forget along sequence length per input, which is what earlier fixed-parameter SSMs (S4) could not do. Selectivity rules out the fast causal convolution used before, so decoding uses a hardware-aware parallel scan rather than a naive sequential recurrence. The block is a single residual path with no attention and no MLP: essentially Mamba block = gated conv1d shortcut + selective SSM, pre-norm residual. The state is a fixed-size (batch, d_inner, d_state) tensor per layer, CONSTANT in sequence length, so there is NO KV cache at all: per-token decode cost is O(d_model * d_state) state bytes instead of O(2 * L * d_h * d_k) KV bytes. That is the defining serving consequence and the reason long-context SSM decoding does not degrade with context length. speedup_reported: the retired record's text was materially fuller (538 chars vs 408) and is RETAINED here rather than discarded. Its text: 5x higher inference throughput than Transformers, against the paper's own GPT-3-style Transformer baselines at matched size; Mamba-3B matches Transformers twice its size in pretraining and downstream evaluation, and performance improves up to million-token sequences. The throughput figure is the authors' own reference implementation on A100 GPUs with their custom kernels and is not a tuned-engine number; published 5x claims for Mamba have consistently shrunk once engines run the model with generic PyTorch rather than the fused scan. AFFILIATIONS: left as-is on this record; the sweep verified it against the paper. VENUE AND THE CLAIMS THAT COMPETED FOR IT: COLM 2024 - SETTLED, and the bait it replaces is named here so it cannot be re-raised. The venue oracle returns THREE records for this exact title with the same two authors: 'CoRR 2023' / dblp.org/journals/CORR/2023 (the arXiv deposit), 'COLM' / colmweb.org/COLM/2024/Conference (the ACCEPTANCE), and 'Submitted to ICLR 2024' / ICLR.cc/2024/Conference/Rejected_Submission (a REJECTION). Mamba is widely believed to be an ICLR paper precisely BECAUSE the ICLR submission is the one that surfaces - a rejection is indexed under a conference venueid and reads like an acceptance unless the Rejected_Submission suffix is checked. Anyone citing 'Mamba, ICLR 2024' is citing a rejection. The peer-reviewed venue is COLM 2024, confirmed by the acceptance venueid in w6p-openreview-api2-duplicate-venue-records. The retired twin mamba-selective-ssm carried venue=arxiv-preprint with year 2023 and a note flagging the pair's disagreement while explicitly deferring the fix; that deferral is discharged here, because there is now one record holding the accepted value. status returns to VERIFIED. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.