Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

Layer-wise interleaving of Mamba with SwiGLU and sliding-window attention. Section 2.1.2: 'Our SWA layer operates on a window size w=2048 that slides over the input sequence, ensuring that the computational complexity remains linear with respect to the sequence length', with RoPE base 10000, FlashAttention 2, and the window size chosen because 'FlashAttention 2 has the same training speed as Mamba's selective parallel scan at the sequence length of 2048'. Reports 3.73x higher throughput than GQA Transformers for 128K user prompts and 3.64x speedup generating 64K tokens with unlimited streaming.