Gemma 3 Technical Report

Section 3: 'we interleave multiple local layers between each global layer, and assign a smaller span of only 1024 tokens to the local layers. Therefore, only the global layers attend to long context, and we have 1 global for every 5 local layers.' Local:Global 1:1 was Gemma 2; 5:1 is Gemma 3. Section 5.2 measured KV-cache effect at 32k context: 'global only' gives 60% memory overhead, reduced to less than 15% with 1:3 and sliding window 1024. Abations show perplexity is insensitive to the ratio down to 7:1 and to window size.