SSM / linear-RNN decode: the constant-size recurrent state update that replaces the KV cache entirely

This class has no KV cache at all, which inverts the entire decode cost model. Per layer the state is S = d_inner * N elements (d_inner = expand*hidden, N = d_state) and each decode step does ONE state update: for Mamba-style, flops = 2*(d_inner*N*2 + d_inner*N) for the B_t@C_t product and the state write, plus the input projection; bytes = 2*d_inner*N*2 (read state, write state). So AI ~= 1 flop/byte on the state alone - it is a weight-class bandwidth cost, not an attention-class one - but the state is CONSTANT in context length, so unlike KV attention its byte count never grows. Worked example from NVIDIA Nemotron-H-8B's config.json (hidden 4096, expand 2 -> d_inner 8192, ssm_state_size 128): state = 8192*128 = 1,048,576 elements = 2.0 MiB per layer at bf16. With 48 Mamba-2 layers and 4 attention layers, resident recurrent state = 96.0 MiB per sequence, independent of context. Compare the 4 attention layers: 4*2*8*128*2 = 16,384 bytes/token = 16 KiB/token, which at 8k context is 128 MiB and already exceeds the entire recurrent state; at 128k it is 2.0 GiB. Nemotron-H states the design intent exactly: Mamba layers 'perform constant computation and require constant memory per generated token'. The corollary nobody quotes: the state is NOT free, it is a fixed per-sequence tax that stops amortising as batch grows, so at very large batch a pure-attention model with paged KV can hold more total tokens than a hybrid with 96 MiB/seq of undividable state.

What SSM/recurrent architectures actually change: they delete a class rather than optimise it. KV-cache growth in context length - the thing that makes long-context decode expensive and that paged attention, GQA, MLA and sliding windows all attack - simply does not exist for these layers. The catch is that they do not delete the attention layers, they delete MOST of them, so the residual attention layers keep the quadratic-ish prefill cost for their share of the model and keep a (small) linear KV cache. Verified hybrid ratios from configs read 2026-10-03: Nemotron-H-8B 48 Mamba / 4 attention (paper: 'Roughly 8% of the total layers in the model are self-attention layers'); Granite-4.0-h-tiny 36 mamba / 4 attention; Jamba states an 8x smaller KV cache than a vanilla Transformer of comparable size purely by trading attention layers for Mamba layers; IBM also reports Jamba-1.5-Large at 94B active parameters. Second-order consequence for serving: the fixed state is a per-sequence cost that never amortises, so a hybrid at batch 256 pays 256*96 MiB of state on Nemotron-H-8B whether the sequences are 1k or 128k tokens. That is the opposite trade from KV cache, which grows with context, and it means the optimal batch size for a hybrid model is set by the state footprint, not by available KV memory. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.