Falcon Mamba: The First Competitive Attention-free 7B Language Model

Pure Mamba (no attention layers at all), 64 layers, hidden 4096 per config.json. Claims significant inference speed and substantially less memory for long-sequence generation due to the architecture alone. Useful as the 'attention-free' end of the hybrid range.