internlm/Intern-S2-397B config.json

Fetched 2026-10-18. Top-level model_type qwen3_5_moe, architectures Qwen3_5MoeForConditionalGeneration, so this is the Qwen3.5-MoE multimodal stack rather than a new architecture. text_config model_type qwen3_5_moe_text: hidden_size 4096, 60 layers, layer_types 45 linear_attention and 15 full_attention with full_attention_interval 4, num_attention_heads 32, num_key_value_heads 2, head_dim 256. Gated linear attention: linear_num_key_heads 16, linear_num_value_heads 64, linear_key_head_dim 128, linear_value_head_dim 128, linear_conv_kernel_dim 4. MoE: num_experts 512, num_experts_per_tok 10, moe_intermediate_size 1024, shared_expert_intermediate_size 1024, mlp_only_layers empty so every layer is MoE. mtp_num_hidden_layers 1. rope_parameters: rope_type default, rope_theta 10000000, partial_rotary_factor 0.25, mrope_interleaved true, mrope_section [11, 11, 10], no rope_scaling key, max_position_embeddings 262144, vocab_size 251392, hidden_act silu. vision_config: hidden 1152, depth 27, 16 heads, patch 16, out_hidden_size 4096. Safetensors total 403,423,094,768 (BF16 403,423,086,128 plus F32 8,640). The geometry matches models/qwen3-5-397b-a17b field for field; this file is the source for the expert count, which the card never states.