Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (arXiv:2402.19427)

arXiv:2402.19427. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. By Soham De et al. (Google DeepMind). Submitted 2024-02-29. Hybrid architecture combining gated linear recurrences (Hawk) with local attention (Griffin). 2026-10-10: Fetched https://arxiv.org/abs/2402.19427. Title: 'Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models'. Preprint matching record claims; abstract confirms hybrid architecture combining gated linear recurrences (Hawk) with local attention (Griffin). 2026-10-18: Read the HTML rendering of the paper. Appendix C Table 2, 14B row: model width D = 5120, RNN width D_RNN = 8192, depth N = 40, MLP expansion factor 3, 40 attention heads, 300B training tokens. Section 2.3: head dimension fixed at 128 with H * 128 = D, attention is multi-query (one KV head), RoPE and no absolute positional embeddings. Section 3.3: Griffin alternates two recurrent residual blocks with one local-MQA residual block, local window fixed at 1024 tokens. Section 2.3 also names GeLU on the recurrent block's second branch. Appendix D describes the implementation as custom JAX with a Pallas scan kernel, measured on TPU-v3.