Gemma 4 26B A4B (MoE, multimodal)

SHARED EXPERT: left null deliberately. config.json declares enable_moe_block true, num_experts 128, top_k_experts 8, moe_intermediate_size 704, and has NO n_shared_experts key at all. Absence of the key is evidence there is no always-on shared expert, but it is an inference from a missing field rather than a positive statement, so this record records null instead of false. Contrast gemma-4-12b / gemma-4-31b, which declare enable_moe_block FALSE and num_experts null - the dense members of the family, correctly leaving the expert fields null as a dense model should. KV GEOMETRY, NOW DERIVED (was null). Gemma 4 uses TWO head geometries in one stack, so the single-formula field is the sum of two terms, both explicit in config: sliding layers = num_layers_sliding x num_key_value_heads x head_dim x 2 x bytes; full layers = num_layers_full x num_global_key_value_heads x global_head_dim x 2 x bytes. DERIVATION for Gemma 4 26B A4B: 25 sliding layers x 8 kv_heads x 256 head_dim x 2 (K+V) x 2 bytes (bf16) = 204,800 B PLUS 5 full-attention layers x 2 global kv_heads x 512 global_head_dim x 2 x 2 = 20,480 B, TOTAL 225,280 B/token (220.0 KiB). layer_types enumerates the split explicitly (25 sliding_attention + 5 full_attention, full at 0-based indices 5,11,17,23,29). attention_k_eq_v is true and num_kv_shared_layers is 0, so no layer shares cache with a neighbour. PARAMS: 25.81B is the safetensors weight count (26B nominal), including the vision tower. ACTIVE: 4B is Google's published A4B figure. Gemma 4 uses TWO head geometries in the same stack: sliding-window layers have head_dim 256 with 8 KV heads, while global layers have global_head_dim 512 with only 2 global KV heads (num_global_key_value_heads). Per global layer that is 2 * 512 = 1024 KV elements versus 8 * 256 = 2048 for a sliding layer, and there are 5 global layers out of 30. A correct figure needs both geometries and the PLE/vision allocation, which config.json does not fully resolve. Deriving a single number here would be a guess. ROUTING: num_experts 128, top_k_experts 8. Eight of 128 is a higher active fraction than gpt-oss's 4 of 128, so routing compute is a bigger share of per-token FLOPs than in gpt-oss despite the smaller active count. moe_intermediate_size 704 against a 2816 hidden size makes each expert very narrow. MEMORY IS SET BY TOTAL, NOT ACTIVE: Google's own doc states all 26B must be resident for fast routing, and puts the BF16 footprint at 57.7 GB (28.8 GB at SFP8, 14.4 GB at Q4_0). active_params_b=4 therefore describes compute per token, not memory. This is the same trap as gpt-oss-120b, stated explicitly by the vendor. attention_k_eq_v=true means K and V share storage in the global layers, which is a real KV reduction independent of head count. final_logit_softcapping=30.0 is retained from Gemma 2. All Gemma 4 sizes ship an MTP draft model for speculative decoding (Google's doc), which changes the decode FLOP mix. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=sliding-window. REQUIRED KERNEL CLASS: a local/windowed attention decode kernel plus per-layer-type cache management. The kernel requirement is not cosmetic: flop/sliding-window-hybrid-attention records that a local layer's decode cost is constant in context (it reads exactly w KV entries) so its arithmetic intensity stays at ~1 flop/byte forever, while the global layers' bytes still grow linearly - the two layer classes have opposite scaling and one kernel must serve both. SUPPORTED BY (cited): vLLM lists sliding attention inside its fallback envelope - 'Attention types: full attention and/or sliding attention' - but ALSO has native rows for the specific SWA architectures: GptOssForCausalLM, Gemma3ForCausalLM, Gemma4ForCausalLM, Olmo3ForCausalLM, GraniteMoeSWAForCausalLM, GraniteSWAForCausalLM [[mdl2-vllm-supported-models-2026-10]]. SGLang's generative table names GPT-OSS as 'hybrid sliding-window-attention MoE model ... with per-layer-type RoPE (YARN on full-attention layers, default on sliding-attention layers) and a softplus per-head attention gate', and lists granite-swash SWA models [[mdl2-sglang-generative-models]]. llama.cpp exposes SWA as first-class KV keys rather than a flag: %%s.attention.sliding_window and %%s.attention.sliding_window_pattern exist as distinct keys per architecture class, alongside SWA-specific width keys, which is direct evidence that SWA changes the cache layout rather than adding an attention flag [[mdl2-llamacpp-arch-class-registry]]. THE PER-CELL CONSEQUENCE: an engine WITHOUT local-attention support does not fail, it silently allocates a full-context cache for every layer. docs/13-model-compatibility.md tabulates this as the KV overpay table; for this model the naive figure in kv_cache_bytes_per_token is the worst case and the effective figure is what the windowed layers actually hold. TWO HARD GATES THAT ARE NOT ABOUT WINDOW SIZE. (1) TensorRT-LLM publishes a per-architecture 'Sliding Window Attention' column and it is NOT uniformly Yes: Yes for DeepseekV4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration and Step3p7ForConditionalGeneration, and explicitly No for Qwen3NextForCausalLM, while the DeepSeek-V3/V3.2/GLM/GPT-OSS/Llama-4/Qwen3.5/Kimi-K3/Nemotron-H rows read N/A [[mdl2-trtllm-supported-models]]. (2) On B200 with vLLM 0.23.1rc1.dev843 + flashinfer-python 0.6.14, a 50-of-60-layer SWA Gemma-4 checkpoint with --kv-cache-dtype fp8_e4m3 and MTP spec-decode (k=7) produces fluent output containing sporadic wrong tokens and short repetition loops once seq_len passes the 1024-token window, because flashinfer's trtllm spec-as-decode kernel mis-computes sliding-window attention for BF16-Q + FP8-KV at certain seq_len/page alignments and vLLM's scale bookkeeping double-applies k/v scales - gotcha-flashinfer-bf16q-fp8kv-spec-decode-corrupts-swa, severity BLOCKER. FP8-Q + spec is clean; BF16-Q + no spec is clean; either drop quantized KV, drop spec decode, or drop the disable_flashinfer_q_quantization flag. This is the cleanest statement in the repo that quantized KV and speculative decoding are not independently safe on an SWA model. (3) FlashInfer also rejects large head sizes at engine init, which is a separate SM120 gap from the page-block one: head_size not supported for gemma-4-31b-it's head_dim 256 / global_head_dim 512 on RTX PRO 6000 [[flashinfer-rejects-large-head-dim]]. (4) On Hopper, vLLM 0.30.0's vllm_flash_attn silently ignores num_splits on SM90, costing 48% per-token decode at batch 1 for Gemma-4 26B-A4B-it global layers (1 query token, 2 KV heads, 2 of an H100's 132 SMs with work), with TRITON_ATTN recovering ~70% [[fa4-num-splits-ignored-sm90]]. (5) vLLM's own scheduler granularity interacts with SWA groups: issue #53786 asks for fine-grained prefix hits for sliding-window groups, and hybrid reconciliation already caps the combined hit at the shorter boundary [[eagle-prefix-cache-last-block-drop]]. (6) On Apple Silicon the class is a cache-CLASS problem, not a kernel problem: mlx-lm's sliding-window layers use RotatingKVCache, which had no quantized implementation, so --kv-bits passes a hasattr presence check and raises NotImplementedError on the first request instead of at startup; the fix (PR 1584) adds RotatingQuantizedKVCache, and the commonly-recommended monkeypatch silently gives up the max_kv_size window bound and yields an unbounded cache [[mlx-kv-bits-crashes-on-first-request-for-hybrid-attention]] [[mdl2-mlxlm-pr1584-rotating-quantized-kv]]. KV OVERPAY NOT COMPUTABLE FROM THIS RECORD for the same reason as gemma-4-12b: separate global head geometry (global_head_dim 512, 2 global KV heads) against local head_dim 256 with 8 KV heads, and kv_cache_bytes_per_token is null. This is also the checkpoint behind two cited engine-level findings: the FlashInfer head_size rejection at head_dim 256 / global_head_dim 512 on SM120 [[flashinfer-rejects-large-head-dim]], and the SM90 num_splits regression costing 48% per-token decode on its global layers at batch 1 [[fa4-num-splits-ignored-sm90]]. The engine cost here is decode throughput, not cache size. CONTEXT_SCALING_METHOD (native): SCHEME RECLASSIFIED (2026-10-04): was 'per-layer-type rope' read as a scaling method. per-layer-type rope_parameters with no factor; 262144 reached natively. LAYER LAYOUT (gemma-4-26b-a4b): layer_types read verbatim from config, one entry per layer; attention_layer_indices are the 0-based positions of the full-attention entries.