Based-1.3B

GEOMETRY FILLED from config.json, and this model has three different mixers, not two. n_layer 36, n_embd 1792, n_head 16, rotary_emb_fraction 1. head_dim is NOT in config; it is n_embd/n_head = 1792/16 = 112. The three mixers, by config key: mixer = BaseConvWithSiLU4 (short causal conv, kernel_sizes [3], expand_proj 4, l_max 2048) is the DEFAULT and applies to the majority of layers; alt_mixer = LinearAttention (feature_dim 16, feature_name taylor_exp, num_heads 16, num_key_value_heads 16, l_max 2048) on alt_mixer_layers [1,6,11,16,21,27,33] = 7 layers; alt_mixer_2 = SlidingsMHA (num_heads 16, window_size 128, causal true) on alt_mixer_2_layers [2,7,12,17,22,28,34] = 7 layers. So the linear attention and the sliding attention are ADJACENT, not interleaved at the same stride: each pair of (1,2), (6,7), (11,12), (16,17), (21,22), (27,28), (33,34). kv_cache_bytes_per_token = 7 SlidingsMHA layers x 16 kv_heads x head_dim 112 x 2 (K+V) x 2 bytes (bf16) = 50,176 B/token (49.0 KiB). Only the 7 sliding-MHA layers carry KV; the 7 linear-attention layers recompute instead of caching, and the 22 short-conv layers have no attention state at all. num_kv_heads 16 / num_heads 16 gives gqa_ratio 1.0. NOTE max_position_embeddings in config is 0 (meaningless); the real bound comes from the mixers' own l_max 2048, so the record carries 2048 on that basis rather than from the top-level field. [SOURCE FIX: this record had an empty sources array and status draft. It now cites config.json. It still needs arXiv:2402.18668 before it can leave draft.] CONTEXT_SCALING_METHOD (native): config declares no rope_scaling and max_position_embeddings 0 (the Based design uses a separate learned length embedding); nothing to scale at the 2048 window the record cites. LAYER LAYOUT (based-1-3b): config has no layer_types; reconstructed from alt_mixer_2_layers (the 7 sliding-window MHA layers, the only ones holding KV) and alt_mixer_layers (the 7 Taylor linear-attention layers); all remaining layers are BaseConv short convolution.