hazyresearch/based-1.3b config.json

Based/SLiM attention model. n_embd 1792, n_layer 36, n_head 16, n_inner 3584, rotary_emb_fraction 1, max_position_embeddings 0. THREE mixer types interleaved: mixer = BaseConvWithSiLU4 (short conv, l_max 2048, kernel_sizes [3], expand_proj 4); alt_mixer = LinearAttention (feature_dim 16, feature_name taylor_exp, num_heads 16, num_key_value_heads 16, l_max 2048, train_view linear) on alt_mixer_layers [1,6,11,16,21,27,33]; alt_mixer_2 = SlidingsMHA (num_heads 16, window_size 128, causal true) on alt_mixer_2_layers [2,7,12,17,22,28,34]. head_dim absent; 1792/16 = 112. Read raw 2026-10-04.