NaiveAI/Naive-N0.5-Flash config.json

Fetched 2026-10-18 from the raw config at commit 9c9a11e8d2a6. model_type naive_n05_flash, architectures NaiveN05FlashForCausalLM, custom code via auto_map. hidden_size 4096, num_hidden_layers 48, num_attention_heads 64, num_key_value_heads 4, head_dim 192, v_head_dim 128, intermediate_size 16384, moe_intermediate_size 2048. MoE: n_routed_experts 256, num_experts_per_tok 8, n_shared_experts null, moe_layer_freq is 0 then 47 ones so layer 0 is dense and the other 47 are MoE. hybrid_layer_pattern is 9 zeros (sparse/DSA) and 39 ones (SWA), zeros at indices 0, 5, 11, 17, 23, 29, 35, 41, 47. sliding_window 128, swa_num_key_value_heads 8, swa_head_dim 192, swa_v_head_dim 128. DSA: enable_dsa true, index_top_k 2048, index_n_heads 16, index_n_kv_heads 1, index_head_dim 128, indexer_activation_dtype fp8_e4m3. rope_theta 10000000, partial_rotary_factor 0.334, rope_scaling type default, max_position_embeddings 1048576, hidden_act silu, vocab_size 152576, tie_word_embeddings false, dtype bfloat16. No kv_lora_rank key, so this is GQA, not MLA.