Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

THE key source for the SSM hardware problem. States directly that SSD 'thanks to its reformulation to use matrix multiplication as a subroutine, can exploit specialized matrix multiplication (matmul) units on GPUs, also known as tensor cores. As a result, it is 2-8x faster than Mamba's fused associative scan, which does not leverage matmul units.' Chunkwise decomposition of the semiseparable matrix; 8x larger recurrent state with minimal slowdown; crosses FlashAttention-2 at sequence length 2K and is 6x faster at 16K.