Mixtral 8x7B

CONTEXT SCALING: null is a VERIFIED value, checked against config rather than left blank. config.json (mistralai/Mixtral-8x7B-Instruct-v0.1) declares rope_scaling null, rope_theta 1000000, max_position_embeddings 32768. Verified native. PARAMS: 46.7B is the safetensors count; the paper states 47B. ACTIVE: 13B is the paper's figure. Neither is contested, both are rounding. KV: 32 * 8 * 128 * 2 * 2 bytes = 131,072 bytes/token = 128 KiB/token, identical per token to Llama 3.1 8B despite being 5.8x the weights. That is the canonical MoE serving point: KV scales with the dense attention tower, not with the expert weights. No sliding window, so the naive KV figure is the real one here. Routed experts with no shared expert. The 2-of-8 ratio means at batch 1 decode a token touches 2 experts, which is friendlier to batching than top-1 routing and much friendlier than 8-of-128. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=dense-gqa. REQUIRED KERNEL CLASS: ordinary causal attention plus GEMM/GEMV. This is the baseline every engine in this repo implements, so the join here is a floor rather than a filter: no engine record in data/engines/ states that dense GQA is unsupported. SUPPORTED BY (cited): vLLM LlamaForCausalLM / Qwen2 / Qwen3 / Mistral / Phi3ForCausalLM rows [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM LlamaForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, Phi3ForCausalLM rows [[mdl2-trtllm-supported-models]]; SGLang generative table (Llama, Qwen, Mistral families) [[mdl2-sglang-generative-models]]; llama.cpp LLM_ARCH_LLAMA [[pap-arch-llamacpp-arch-registry]]. HARD CONSTRAINTS THAT DO APPLY even to dense GQA, and they are model-shape rather than architecture-class: tensor-parallel size must divide the attention head count or the engine refuses to start [[mgpu-tp-size-must-divide-attention-heads]] (affects vllm and sglang); and a GQA KV cache is the thing paged-attention machinery exists for, so an engine with no paged KV (llama.cpp's slot-based pool [[llama-cpp]], exllamav2 [[exllamav2]]) loses the concurrency lever rather than the ability to run. ONE HARDWARE-DEPENDENT KERNEL GATE THAT IS NOT ABOUT GQA: on RDNA, vLLM's custom paged-attention kernel is gated on head_size == 128 AND block_size == 16, so a dense GQA model with head_dim 256 falls to the Triton 2D fallback and decode falls off a cliff - measured 12.1 tok/s at 518 tokens of context to 4.2 tok/s at 32k, with kernel_paged_attention_2d growing 28.3x [[rdna-triton-paged-attn-decode-cliff]]. Check head_dim before blaming the card.