Gemma 2 model card and HF config: alternating local/global attention, sliding_window 4096
repo
Fields
id
sources/gemma2-report-alternating-swaPublisher
Google / Hugging Face
Kind
repo
Published
2024-06
Accessed
2026-10-18
notes
vendor_claim for the layer pattern, used together with the catalog snapshot model-fr-hf-config-gemma-2-9b. The Gemma 2 report and the HF model card state a 1:1 interleaving of local sliding-window layers (window 4096) and global layers. The catalog model record already stores sliding_window 4096, 42 layers, 8 kv heads, head_dim 256, and the naive kv_cache_bytes_per_token of 344064, and notes that real resident KV is about half the naive figure. This source exists so the derived effective-KV record can cite the pattern without inventing a layer count the config does not enumerate.
Notes
vendor_claim for the layer pattern, used together with the catalog snapshot model-fr-hf-config-gemma-2-9b. The Gemma 2 report and the HF model card state a 1:1 interleaving of local sliding-window layers (window 4096) and global layers. The catalog model record already stores sliding_window 4096, 42 layers, 8 kv heads, head_dim 256, and the naive kv_cache_bytes_per_token of 344064, and notes that real resident KV is about half the naive figure. This source exists so the derived effective-KV record can cite the pattern without inventing a layer count the config does not enumerate.
Cross-references
This record references no other records.