Jamba: A Hybrid Transformer-Mamba Language Model

Interleaves Transformer and Mamba layers with MoE. States the design goal explicitly: 'Trading off attention layers for Mamba layers reduces the total size of the KV cache. Our architecture aims to provide not only a small number of active parameters but also an 8x smaller KV cache compared to a vanilla Transformer.' Released config: 52B total / 12B active, fits a single 80GB GPU.