inference/research
verifiedhigh · 0.85updated 2026-10-18

Gemma 2 model card and HF config: alternating local/global attention, sliding_window 4096

repo

Fields

id
sources/gemma2-report-alternating-swa
Publisher
Google / Hugging Face
Kind
repo
Published
2024-06
Accessed
2026-10-18
notes
vendor_claim for the layer pattern, used together with the catalog snapshot model-fr-hf-config-gemma-2-9b. The Gemma 2 report and the HF model card state a 1:1 interleaving of local sliding-window layers (window 4096) and global layers. The catalog model record already stores sliding_window 4096, 42 layers, 8 kv heads, head_dim 256, and the naive kv_cache_bytes_per_token of 344064, and notes that real resident KV is about half the naive figure. This source exists so the derived effective-KV record can cite the pattern without inventing a layer count the config does not enumerate.

Notes

vendor_claim for the layer pattern, used together with the catalog snapshot model-fr-hf-config-gemma-2-9b. The Gemma 2 report and the HF model card state a 1:1 interleaving of local sliding-window layers (window 4096) and global layers. The catalog model record already stores sliding_window 4096, 42 layers, 8 kv heads, head_dim 256, and the naive kv_cache_bytes_per_token of 344064, and notes that real resident KV is about half the naive figure. This source exists so the derived effective-KV record can cite the pattern without inventing a layer count the config does not enumerate.

Cross-references

This record references no other records.