Gemma 3 4B IT (multimodal)

TWO-PART MODEL, and the vision side is the most expensive-per-image design in this slice. (1) Vision: SigLIP tower, 27 layers, hidden 1152, 16 heads, patch 14, image_size 896, vision_use_head false, gelu_pytorch_tanh. NO POOLING, NO MERGE, NO RESAMPLER. VISION TOKENS = (896/14)^2 = 64*64 = 4,096 tokens for ONE image -- the largest single-image vision token count of any model here, 64x SmolVLM's 64 and 2x Pixtral's single 1024 tile. Gemma 3 compensates with pan-and-scan (optional multi-crop at inference), which multiplies the 4,096 figure. DOMINANCE: VISION DOMINATES ABSOLUTELY. 4,096 vision tokens against a typical 100-300 token text prompt is 15:1 to 40:1. On any single-image Gemma 3 request, the SigLIP forward pass and the LLM prefill over 4,096 image tokens -- not the decode loop -- are the entire cost. (2) LLM: Gemma 3 text decoder, 34 layers, hidden 2560, 8 heads / 4 KV heads, head_dim 256 (note: head_dim 256 is 4x the hidden_size/num_heads of 320, so it decouples head width from model width), intermediate 10240, query_pre_attn_scalar 256, vocab 262,208, max_position 131,072. ATTENTION: sliding_window 1024 with 5-local-to-1-global layer interleaving -- the sliding layers cap attention cost while the global layers preserve long-range. ARCHITECTURE = dense (both parts dense). CONTEXT_SCALING: rope_scaling {factor 8.0, rope_type linear} on top of rope_theta 1e6 -- simple linear position interpolation, not YaRN or NTK-aware. The 131,072 figure is therefore an extrapolated context, not a natively trained one. KV CACHE: 34 * 4 * 256 * 2 * 2 = 139,264 B/token (136 KiB). IMPORTANT: in a real Gemma 3 serving stack only the 6 global-attention layers hold full-length KV; the 28 sliding layers hold at most 1,024 tokens each. A correct paged-KV implementation should therefore store roughly 34 -> ~6 full layers plus 28 capped, cutting effective resident KV by roughly 5x versus this headline number. This is the single most important serving caveat in this record and it is invisible in config.json alone. params_b 4.300 = HF API safetensors total (4,300,079,472); vision tower is ~0.4B of it. PROVENANCE: google/gemma-3-4b-it is license-gated (raw config.json returns HTTP 401). Every field here was read from the unsloth/gemma-3-4b-it mirror, and the param total was cross-checked against the gated repo's own HF API entry. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=sliding-window. REQUIRED KERNEL CLASS: a local/windowed attention decode kernel plus per-layer-type cache management. The kernel requirement is not cosmetic: flop/sliding-window-hybrid-attention records that a local layer's decode cost is constant in context (it reads exactly w KV entries) so its arithmetic intensity stays at ~1 flop/byte forever, while the global layers' bytes still grow linearly - the two layer classes have opposite scaling and one kernel must serve both. SUPPORTED BY (cited): vLLM lists sliding attention inside its fallback envelope - 'Attention types: full attention and/or sliding attention' - but ALSO has native rows for the specific SWA architectures: GptOssForCausalLM, Gemma3ForCausalLM, Gemma4ForCausalLM, Olmo3ForCausalLM, GraniteMoeSWAForCausalLM, GraniteSWAForCausalLM [[mdl2-vllm-supported-models-2026-10]]. SGLang's generative table names GPT-OSS as 'hybrid sliding-window-attention MoE model ... with per-layer-type RoPE (YARN on full-attention layers, default on sliding-attention layers) and a softplus per-head attention gate', and lists granite-swash SWA models [[mdl2-sglang-generative-models]]. llama.cpp exposes SWA as first-class KV keys rather than a flag: %%s.attention.sliding_window and %%s.attention.sliding_window_pattern exist as distinct keys per architecture class, alongside SWA-specific width keys, which is direct evidence that SWA changes the cache layout rather than adding an attention flag [[mdl2-llamacpp-arch-class-registry]]. THE PER-CELL CONSEQUENCE: an engine WITHOUT local-attention support does not fail, it silently allocates a full-context cache for every layer. docs/13-model-compatibility.md tabulates this as the KV overpay table; for this model the naive figure in kv_cache_bytes_per_token is the worst case and the effective figure is what the windowed layers actually hold. TWO HARD GATES THAT ARE NOT ABOUT WINDOW SIZE. (1) TensorRT-LLM publishes a per-architecture 'Sliding Window Attention' column and it is NOT uniformly Yes: Yes for DeepseekV4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration and Step3p7ForConditionalGeneration, and explicitly No for Qwen3NextForCausalLM, while the DeepSeek-V3/V3.2/GLM/GPT-OSS/Llama-4/Qwen3.5/Kimi-K3/Nemotron-H rows read N/A [[mdl2-trtllm-supported-models]]. (2) On B200 with vLLM 0.23.1rc1.dev843 + flashinfer-python 0.6.14, a 50-of-60-layer SWA Gemma-4 checkpoint with --kv-cache-dtype fp8_e4m3 and MTP spec-decode (k=7) produces fluent output containing sporadic wrong tokens and short repetition loops once seq_len passes the 1024-token window, because flashinfer's trtllm spec-as-decode kernel mis-computes sliding-window attention for BF16-Q + FP8-KV at certain seq_len/page alignments and vLLM's scale bookkeeping double-applies k/v scales - gotcha-flashinfer-bf16q-fp8kv-spec-decode-corrupts-swa, severity BLOCKER. FP8-Q + spec is clean; BF16-Q + no spec is clean; either drop quantized KV, drop spec decode, or drop the disable_flashinfer_q_quantization flag. This is the cleanest statement in the repo that quantized KV and speculative decoding are not independently safe on an SWA model. (3) FlashInfer also rejects large head sizes at engine init, which is a separate SM120 gap from the page-block one: head_size not supported for gemma-4-31b-it's head_dim 256 / global_head_dim 512 on RTX PRO 6000 [[flashinfer-rejects-large-head-dim]]. (4) On Hopper, vLLM 0.30.0's vllm_flash_attn silently ignores num_splits on SM90, costing 48% per-token decode at batch 1 for Gemma-4 26B-A4B-it global layers (1 query token, 2 KV heads, 2 of an H100's 132 SMs with work), with TRITON_ATTN recovering ~70% [[fa4-num-splits-ignored-sm90]]. (5) vLLM's own scheduler granularity interacts with SWA groups: issue #53786 asks for fine-grained prefix hits for sliding-window groups, and hybrid reconciliation already caps the combined hit at the shorter boundary [[eagle-prefix-cache-last-block-drop]]. (6) On Apple Silicon the class is a cache-CLASS problem, not a kernel problem: mlx-lm's sliding-window layers use RotatingKVCache, which had no quantized implementation, so --kv-bits passes a hasattr presence check and raises NotImplementedError on the first request instead of at startup; the fix (PR 1584) adds RotatingQuantizedKVCache, and the commonly-recommended monkeypatch silently gives up the max_kv_size window bound and yields an unbounded cache [[mlx-kv-bits-crashes-on-first-request-for-hybrid-attention]] [[mdl2-mlxlm-pr1584-rotating-quantized-kv]]. KV OVERPAY: naive 34*4,096 = 139,264 B/token; effective with local support = (6*4096*c + 28*4096*1024)/c = 24,576 + 117,440,512/c - 5.47x at 131,072 context, 4.95x at 32k, 3.58x at 8k. CONTEXT_SCALING_METHOD (linear): config-derived; the existing free text 'linear RoPE (factor 8)' agrees.