Mamba-Codestral-7B-v0.1

head_dim = 64 is recorded from config, but read it as the Mamba SSM state width, NOT an attention head dim: Mamba-2 configures it as head_dim in the config class for the mamba mixing dimensions. There is no softmax attention in this model, so num_kv_heads, gqa_ratio and context_scaling stay null - they are UNDEFINED, not zero. kv_cache_bytes_per_token is 0 because the recurrent state is O(1) in context length and lives outside the KV path. Pure SSM: config.json declares architectures ['Mamba2ForCausalLM'], model_type 'mamba2', and contains no num_attention_heads, num_key_value_heads or max_position_embeddings fields at all - there is no attention and no positional-embedding limit in the config. (head_dim 64 in config is the Mamba head dimension, not an attention head_dim, so it is left null here.) kv_cache_bytes_per_token = 0: a Mamba-2 layer carries a fixed-size recurrent state instead of a per-token KV cache, so nothing grows with sequence length. The state is not free, it is constant: with state_size 128, head_dim 64 and num_heads 128 the per-layer recurrent state is 128 heads x 64 x 128 = 1,048,576 elements = 2.0 MiB at bf16, x 64 layers = 128 MiB per sequence, independent of context length. (n_groups 8 is how many heads share each SSM group for the scan, not a multiplier on state size.) For a pure-SSM model that 128 MiB is the entire per-sequence state, against 0 for KV - which is the sharpest illustration in this slice of what 'no KV cache' actually means: not zero memory, but a fixed cost that scales with concurrency and not with context. conv1d with conv_kernel 4 adds a short fixed convolution state on top. Serving shape: 64 layers at hidden 4096 with intermediate_size only 8192, so the model is depth-heavy and MLP-light compared with a 7B dense transformer; compute per token is spread over a scan that parallelises over chunk_size 256, which changes the arithmetic-intensity profile completely - it is not a GEMM-bound decode. This is the architecture llama.cpp and vLLM need their dedicated mamba kernels for. context_scaling is null: context length is native to the pretrained config, no rope_scaling. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=ssm. REQUIRED KERNEL CLASS: two kernel families in one graph - a chunked linear-attention / delta-rule kernel for the SSM or linear-attention layers, and an ordinary or MLA decode kernel for the full-attention minority - plus a fixed-size recurrent state allocator. Three independent records say the recurrent branch is the hard part: flop/ssm-recurrent-state-update (the state is a fixed per-sequence tax that never amortises, so it scales with concurrency and NOT with context); flop/ssm-sequential-scan-tensor-core-mismatch (the recurrence is a multiplicative chain that tensor cores cannot express, so it must run as a sequential scan; the published SSD-vs-fused-scan gap is 2-8x for identical mathematics); and llama.cpp's own dispatch table, which puts every hybrid and SSM architecture in its supports_sm_tensor == false list [[mdl2-llamacpp-arch-class-registry]] - the engine agreeing with the arithmetic. That same list also shows llama.cpp needed ONE shared hybrid code path (llm_arch_is_hybrid, 18 entries: JAMBA, FALCON_H1, PLAMO2, GRANITE_HYBRID, LFM2, LFM2MOE, NEMOTRON_H, NEMOTRON_H_MOE, QWEN3NEXT, KIMI_LINEAR, BAILINGMOE3, KIMI_K3, GLM5_NEXT, QWEN35, QWEN35MOE, QWEN4EXP, DEEPSEEK4, MINIMAX_01) covering exactly the families vLLM and SGLang each ship as separate rows, which is why hybrid support arrives per engine release rather than at once. SUPPORTED BY (cited): vLLM native rows MambaForCausalLM, Mamba2ForCausalLM, JambaForCausalLM, FalconMambaForCausalLM, FalconH1ForCausalLM, NemotronHForCausalLM, GraniteMoeHybridForCausalLM, Qwen3NextForCausalLM, KimiLinearForCausalLM, Lfm2ForCausalLM, Lfm2MoeForCausalLM, OlmoHybridForCausalLM, KimiK3ForConditionalGeneration, BailingMoeV3ForCausalLM, Qwen3_5 rows [[mdl2-vllm-supported-models-2026-10]]; SGLang generative table names Granite 4.0 micro as 'hybrid Mamba-MoE ... combining gated short convolutions with a small number of grouped query attention blocks', Kimi Linear as 'hybrid linear attention model ... Kimi Delta Attention (KDA) for up to 6x faster decoding and 75% KV cache reduction vs full attention', Nemotron Nano as 'hybrid Mamba-Transformer architecture', Ling as 'hybrid MoE model: per-layer mix of DeepSeek-style MLA full attention and Qwen3 Gated-Delta-Net (GDN) linear attention', LFM2 as 'hybrid backbone', Mamba-Codestral as 'Mamba2 ... holds no KV cache', and its multimodal page carries the descriptor 'hybrid KDA/MLA' [[mdl2-sglang-generative-models]] [[mdl2-sglang-multimodal-language-models]]. TensorRT-LLM has rows for NemotronHForCausalLM, Qwen3NextForCausalLM, Qwen3_5MoeForCausalLM, Qwen4ExpForCausalLM, KimiLinearForCausalLM and KimiK3ForConditionalGeneration but NO row at all for MambaForCausalLM, Mamba2ForCausalLM, FalconH1ForCausalLM, Lfm2ForCausalLM or GraniteMoeHybridForCausalLM [[mdl2-trtllm-supported-models]]. PER-FAMILY FEATURE TRAPS, all from the TRT-LLM key-models matrix, which is the only per-architecture feature matrix in this repo [[mdl2-trtllm-supported-models]]: Qwen3NextForCausalLM has Sliding Window Attention = No and Speculative Decoding = No; Kimi K3 has no MTP or EAGLE-3 head at all (spec method DSpark only) and needs Blackwell SM100; MiniMax-M3 sparse-attention path supports neither KV cache reuse nor MTP in this release; DeepSeek-V4 and Step-3.7 are 'Untested' for disaggregated serving. KERNEL BUGS THAT ARE ARCHITECTURE-SPECIFIC AND REPRODUCED: (1) vLLM's EAGLE last-block drop (drop_eagle_block) costs 6-10 points of prefix-cache hit rate, and on a hybrid GDN/Mamba layout the scheduler alignment unit is 1,648 tokens instead of a 16-token dense block, so the same protection costs ~100x more cache hit - 97.5% to 86.9-91.7% hit rate and 30-40% batch throughput on prefix-reusing workloads [[eagle-prefix-cache-last-block-drop]]. (2) llama.cpp decode collapses on hybrid gated-deltanet GGUF models at high KV position: 33 t/s at 68K to 1.4 t/s at 91K with the model fully resident, across KV quant types and with FA on or off, reproduced on Windows and WSL2 on master and release [[gated-deltanet-decode-collapse-long-context]]. (3) llama.cpp's SYCL path has a regression specific to hybrid linear-attention graphs - qwen3next/qwen3_35 architecture models crash in ggml_sycl_op_mul_mat or return empty output on 2x Arc Pro B60, bracketed between builds b8477 (works) and b9479 (fails), while dense-attention models on the same cards are unaffected [[sycl-hybrid-linear-attention-arch-crash-intel-arc]]. (4) FlashInfer rejects large head sizes at engine init for the head geometry these hybrids introduce [[flashinfer-rejects-large-head-dim]]. PURE SSM SPECIFICS (this architecture has NO attention layers at all): the required kernel is the chunked SSD / state-space-duality scan and nothing else - there is no attention kernel to select. flop/ssm-sequential-scan-tensor-core-mismatch carries the published penalty (the matmul-reformulated SSD is 2-8x faster than the fused associative scan for identical mathematics, and Mamba-2 SSD crosses FlashAttention-2 at sequence length 2K), and llama.cpp's dispatch table puts MAMBA and MAMBA2 in llm_arch_is_recurrent and in the supports_sm_tensor == false list, confirming there is no tensor-core path for these layers [[mdl2-llamacpp-arch-class-registry]]. ENGINE JOIN (cited): vLLM has native MambaForCausalLM and Mamba2ForCausalLM rows [[mdl2-vllm-supported-models-2026-10]]; SGLang names mistralai/Mamba-Codestral-7B-v0.1 in its generative table with the mechanism stated - 'Mamba2 state-space model for code generation; uses selective state spaces instead of attention, so it holds no KV cache' [[mdl2-sglang-generative-models]]. TensorRT-LLM has NO Mamba or Mamba2 row in its architecture table at all [[mdl2-trtllm-supported-models]]. llama.cpp reaches it via LLM_ARCH_MAMBA2 [[pap-arch-llamacpp-arch-registry]]. WHAT THIS ARCHITECTURE DOES TO AN ENGINE: kv_cache_bytes_per_token = 0 is correct and the memory is NOT flat - the 128 MiB of Mamba-2 state per sequence is a fixed cost that scales with CONCURRENCY and not with context, which is the opposite trade from KV. flop/ssm-recurrent-state-update states the corollary directly: at very large batch a pure-attention model with paged KV can hold more total tokens than a hybrid with 96 MiB/seq of undividable state. An engine's paged-attention machinery should simply skip these layers, and the state must be allocated contiguously and never evicted because it is fixed-size. CONTEXT_SCALING_METHOD (native): pure Mamba-2, no attention and no positional embeddings at all.