Voxtral-Mini-3B-2507

CONTEXT SCALING: null is a VERIFIED value here, checked against config rather than left blank. config.json (mistralai/Voxtral-Mini-3B-2507) declares rope_scaling null, rope_theta 100000000, max_position_embeddings 131072. Verified null. TWO-PART MODEL, audio-in / text-out (Mistral's ASR model -- it transcribes, it does not speak). This is encoder-decoder in the precompute sense: a Whisper-style AUDIO ENCODER feeds a causal text LLM through a projector, and all decoding happens in the LLM. (1) AUDIO ENCODER (the compute-heavy part): model_type voxtral_encoder, 32 layers, hidden_size 1280, 20 heads AND 20 KV heads (MHA), head_dim 64, intermediate 5120, gelu, num_mel_bins 128, max_source_positions 1500, vocab_size 51866 (the Whisper vocabulary, i.e. the encoder's output space is Whisper-shaped). Preprocessor: WhisperFeatureExtractor, chunk_length 30s, n_samples 480,000, nb_max_frames 3000, sampling_rate 16000, n_fft 400, hop 160, feature_size 128. So exactly the Whisper input pipeline and encoder geometry -- which is what lets Mistral reuse Whisper's audio stack wholesale. AUDIO TOKENS: 1,500 encoder positions per 30s chunk. NO KV CACHE FOR THE ENCODER: single bidirectional pass, nothing cached. Unlike Whisper there is also no separate cross-attention KV -- the projector feeds the LLM as a prefix of tokens, so the 1,500 audio positions enter the LLM's own KV cache like any other prefix. That is a simpler memory story than encoder-decoder cross-attention. (2) LLM: llama dense decoder, 30 layers, hidden 3072, 32 heads / 8 KV heads, head_dim 128, intermediate 8192, vocab 131,072, max_position 131,072, rope_theta 1e8, rope_scaling null, attention_bias false, tie via pretraining_tp 1. projector_hidden_act gelu. ARCHITECTURE = dense. PROJECTOR: a 2-layer MLP (projector_hidden_act) mapping the 1280-dim audio encoder to the 3072-dim LLM. AUDIO DYNAMICS: audio_token_id 24. KV CACHE: 30 * 8 * 128 * 2 * 2 = 122,880 B/token (120 KiB) -- LLM self-attention only; the audio encoder contributes nothing per token. Note that the 1,500 audio tokens DO occupy this cache, so a single 30s transcription request pins 1,500 * 122,880 = 184 MiB of LLM KV before the first output token, growing with each generated token. That prefix cost is inherent to the prefix-projection design and is worth budgeting against Whisper's fixed 234 MiB cross-attention. STREAMING vs OFFLINE: OFFLINE, same as Whisper -- bidirectional 1,500-position encoder over a fixed 30s chunk (preprocessor chunk_length 30). Not streamable without re-architecting the encoder. NAME vs REALITY: the name says 3B but the HF API safetensors total is 4,676,271,104 (4.676B). The gap is the audio encoder plus a large untied embedding: the text body alone is ~3.4B (30 layers x (4*3072^2 attention + 3*3072*8192 MLP) = 30 x 113M) and the two 131,072 x 3072 embedding matrices add 805M. The '3B' marketing figure appears to count the text backbone only. Recorded as 4.676 from tensors, which is authoritative; the naming discrepancy is flagged rather than resolved. flops_per_token_active is NULL rather than 2*params for the same reason as Whisper: the 1,500-position encoder precompute dominates and is not a per-generated-token cost. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=dense-gqa. REQUIRED KERNEL CLASS: ordinary causal attention plus GEMM/GEMV. This is the baseline every engine in this repo implements, so the join here is a floor rather than a filter: no engine record in data/engines/ states that dense GQA is unsupported. SUPPORTED BY (cited): vLLM LlamaForCausalLM / Qwen2 / Qwen3 / Mistral / Phi3ForCausalLM rows [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM LlamaForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, Phi3ForCausalLM rows [[mdl2-trtllm-supported-models]]; SGLang generative table (Llama, Qwen, Mistral families) [[mdl2-sglang-generative-models]]; llama.cpp LLM_ARCH_LLAMA [[pap-arch-llamacpp-arch-registry]]. HARD CONSTRAINTS THAT DO APPLY even to dense GQA, and they are model-shape rather than architecture-class: tensor-parallel size must divide the attention head count or the engine refuses to start [[mgpu-tp-size-must-divide-attention-heads]] (affects vllm and sglang); and a GQA KV cache is the thing paged-attention machinery exists for, so an engine with no paged KV (llama.cpp's slot-based pool [[llama-cpp]], exllamav2 [[exllamav2]]) loses the concurrency lever rather than the ability to run. ONE HARDWARE-DEPENDENT KERNEL GATE THAT IS NOT ABOUT GQA: on RDNA, vLLM's custom paged-attention kernel is gated on head_size == 128 AND block_size == 16, so a dense GQA model with head_dim 256 falls to the Triton 2D fallback and decode falls off a cliff - measured 12.1 tok/s at 518 tokens of context to 4.2 tok/s at 32k, with kernel_paged_attention_2d growing 28.3x [[rdna-triton-paged-attn-decode-cliff]]. Check head_dim before blaming the card. CONTEXT_SCALING_METHOD (native): rope_theta 1e8 with rope_scaling null: a training base, not a scaling method.