id
flops/sliding-window-hybrid-attention
Arithmetic intensity
A local layer replaces the L x L causal attention matrix with L x w, so prefill flops fall by the factor L/w while bytes for K and V fall by the same factor only for the retained window - the cache per token per LOCAL layer is bounded at w and does not grow with context at all. Gemma 3 states the design directly: 'we interleave multiple local layers between each global layer, and assign a smaller span of only 1024 tokens to the local layers. Therefore, only the global layers attend to long context, and we have 1 global for every 5 local layers.' With a 5:1 local:global ratio the model-wide prefill attention cost is (5/6)*L*w + (1/6)*L*L, so at L=131072 and w=1024 the local layers contribute 0.4% of what full attention would - total prefill attention is cut ~5.9x at that length, and the cut grows linearly with L. The residual global fraction is what sets the asymptote, and this is why the ratio is the published knob: Gemma 3's ablation reports 'minimal impact on perplexity when changing this ratio' down to 7:1 local, and separately that a 7:1 local:global ratio has minimal perplexity impact. Because those global layers keep the full quadratic behaviour, hybrid-local attention changes the CONSTANT on prefill cost, never the order. Decode side, the arithmetic inverts cleanly: a local layer's decode cost is constant in context (it reads exactly w KV entries), so AI for the local part stays at ~1 flop/byte forever while the global part's bytes still grow linearly - meaning a 5:1 Gemma-3-shaped model gets ~83% of its decode attention bytes from 1/6 of its layers.
Scales with
sliding_window w (the local cost scales linearly with it)context_length (local layers: no cache growth; global layers: linear cache, quadratic prefill)local:global layer ratio (5:1 Gemma 3, 1:1 Gemma 2, 12:12 GPT-OSS, 1:4 Ministral-3 published pattern)num_global_layers (the asymptotic term - irreducible without dropping global attention entirely)batch_size (local decode AI stays at 1 flop/byte at any batch; it is a fixed cache read)
Workarounds
Interleave a high local:global ratio - Gemma 3 shows perplexity is insensitive down to 7:1 and the cache saving continues, so this is close to free qualityShorten the window rather than the ratio - Gemma 3's own ablation shows the window can be 'reduced significantly without impacting perplexity' (tested 512/1024/2048/4096/8192)Block-sparse / FlashAttention-style kernels so a w-window is a bounded set of tiles rather than a masked full-length scan, which is what makes the local layer cheap rather than merely sparseKeep at least a few global layers: GPT-OSS alternates exactly 1:1 and Kimi Linear keeps periodic full attention specifically 'preserving global information flow via the full attention layers'Cache only the w most recent tokens per local layer; the allocation is static per layer, so paged-KV engines can size local layers independently of context
notes
The published window sizes and ratios, all verified 2026-10-03. Gemma 3: w=1024, 5 local per 1 global, starting with a local layer (config.json confirms sliding_window 1024, sliding_window_pattern 6, and the 5:1 pattern in the technical report); Gemma 2 was 1:1 at w=4096 per the Gemma 3 report. GPT-OSS-20B config.json: 24 layers with layer_types alternating sliding_attention/full_attention, sliding_window 128, 8 KV heads, head_dim 64 - an unusually small window and an unusually high global fraction. Samba: w=2048 sliding-window layers interleaved with Mamba. Kimi Linear: 3:1 KDA:full-attention with config.json listing 20 KDA layers and 7 full attention layers out of 27. Ministral-3-3B config.json reports sliding_window null despite being widely described as sliding-window; I could not verify a local window for it and record it as unpublished. Llama 4 Scout: the gated meta-llama repo was unreadable, the unsloth mirror config exposes NO sliding-window field, so any Llama-4 local window figure is UNVERIFIED here and is not asserted. The measured payoff Gemma 3 publishes: at 32k context, 'global only' configuration gives 60% memory overhead from the KV cache, 'reduced to less than 15% with 1:3 and sliding windows of 1024'. Engineering consequence: sliding-window hybrids are the cheapest available long-context win and the mechanism is different from MLA's - they cut the number of layers that hold a growing cache rather than shrinking the per-layer cache. They also fail in a specific, predictable way: a purely local model loses global recall, so every serious design in this set retains a global fraction, and that fraction is the irreducible part of the cost.