Jamba: A Hybrid Transformer-Mamba Language Model

Pure Mamba models lose the exact recall of attention; pure attention models pay a KV cache that grows with context and cannot fit long contexts in memory.

Interleaves blocks of Transformer and Mamba layers in a fixed repeating ratio (the 52B Jamba-v0.1 configuration uses roughly a 1:3 Mamba:Transformer block ratio) and adds MoE to some layers for capacity at controlled active parameters, so the released model fits on a single 80GB GPU. The layer structure is the practical contribution: enough Mamba layers that most of the model has constant-size state and no KV contribution, plus a minority of full-attention layers that restore exact recall. Load balancing in the MoE layers is the DeepSeekMoE-style fine-grained split with shared experts. Reported strong results to 256K context. Serving consequence: KV cache is proportional to the number of attention layers only, not to total layers, which is the whole economic argument for the hybrid, and this is also where batching gets hard because Mamba state must be copied in and out of the scheduler per request.

arXiv preprint (v2 2024-07-03); no journal-ref on the abs page. The abstract states 'high throughput and small memory footprint compared to vanilla Transformers' with no ratio and no stated baseline configuration, so speedup_reported is null rather than reconstructed. Engine support is solidly verified: vLLM has a native JambaForCausalLM row (AI21-Jamba-1.5-Large, AI21-Jamba-1.5-Mini, Jamba-v0.1), and llama.cpp has LLM_ARCH_JAMBA inside llm_arch_is_hybrid() with a dedicated conversion/jamba.py. Successor paper is Jamba-1.5 (arXiv 2408.12570), recorded separately. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.