Griffin-14B

google/griffin-14b is no longer reachable on the HF hub (a search returns no such repo) and the model card/config is not retrievable, so the Hawk/local-attention geometry could not be verified from primary source. num_kv_heads, head_dim, gqa_ratio and kv_cache_bytes_per_token left null. Note the architecture itself is well documented in arXiv:2402.19427, which is the citation this record already carries. Scaled to 14B parameters per the Griffin paper (arXiv:2402.19427). Hawk (pure RNN with gated linear recurrences) exceeds Mamba's reported downstream performance. Griffin (hybrid) matches Llama-2 performance despite 6x fewer training tokens. 'During inference they have lower latency and significantly higher throughput' (qualitative, no numeric ratio). Weights released but no mainstream inference engine ships a Griffin mode. Research-only adoption. [Source fixed from non-existent 'hawk-griffin-gated-linear-recurrence' to 'pap-eff-hawk-griffin'.] CONTEXT_SCALING_METHOD (native): Hawk/gated-linear-recurrence + local attention: the architecture carries position in the recurrence, not in a rope base. config could not be read (ai2/griffin-14b returns 401), so this is from the architecture, not from config. KV GEOMETRY FILLED 2026-10-18 from the paper, not from a config.json (google/griffin-14b and ai2/griffin-14b are still unreachable). arXiv:2402.19427 Appendix C Table 2 gives the 14B row: model width D = 5120, RNN width D_RNN = 8192, depth N = 40 residual blocks, MLP expansion 3, 40 attention heads. Section 2.3 fixes the head dimension at 128 and sets the head count so H * 128 = D, which is exactly 40 heads at D = 5120, and states the attention is multi-query: one key-value head per attention block. Section 3.3 gives the interleave: two residual blocks with a recurrent block, then one residual block with local MQA, so 1 of every 3 residual blocks is attention: 14 of 40. The same section fixes the local attention window at 1024 tokens unless otherwise stated. The growing KV cache is therefore bounded by the window rather than by context. 14 attention blocks x 1 kv head x 128 head_dim x 2 (K and V) x 2 bytes (bf16) x 1024 tokens = 9,175,040 bytes per sequence, about 8.75 MiB, independent of context length. kv_cache_bytes_per_token is recorded as 0, the same convention models/lfm2-1-2b uses for a hybrid whose cache does not grow with context; the per-sequence window cost is the number above, not a per-token rate. num_kv_heads is 1 because MQA has a single KV head per attention block (gqa_ratio = 40 query heads / 1). activation is GeLU on the recurrent block's second branch (Section 2.3); the MLP block is gated but the paper does not name a separate MLP activation. Still unpublished: the rope base, and any trained context cap beyond the 2048-token training length. ENGINES: none. The paper's implementation is a custom JAX/Pallas stack (Appendix D), and no llama.cpp, vLLM, SGLang or TensorRT-LLM build lists Griffin. There is no expert routing, so expert offload does not apply; the whole 14B is active and the recurrent state plus the 1024-token attention window both stay resident.