ornith-ai/Ornith-1.5-35B-A3B config.json

Fetched 2026-10-18 from resolve/main/config.json, commit 10fbf86fed7ecee4a061f8b499a618f46001cac1. architectures Qwen3_5MoeForConditionalGeneration, model_type qwen3_5_moe. text_config: hidden_size 2048, num_hidden_layers 40, num_attention_heads 16, num_key_value_heads 2, head_dim 256, partial_rotary_factor 0.25, num_experts 256, num_experts_per_tok 8, moe_intermediate_size 512, shared_expert_intermediate_size 512, vocab_size 248320, max_position_embeddings 262144, hidden_act silu, tie_word_embeddings false, mtp_num_hidden_layers 1. layer_types is 30 linear_attention + 10 full_attention at full_attention_interval 4. rope_parameters: rope_type default, rope_theta 10000000, mrope_interleaved true, mrope_section [11, 11, 10]. No rope_scaling key.