reflection/Beam-1.0-501B-A23B config.json

Fetched 2026-10-18 from resolve/main/config.json. architectures BeamForCausalLM, model_type beam. hidden_size 5120, intermediate_size 20480, num_hidden_layers 52, num_attention_heads 48, num_key_value_heads 8, head_dim 128, hidden_act silu, max_position_embeddings 1048576, vocab_size 151936, tie_word_embeddings false. MoE: num_experts 256, num_experts_per_tok 8, moe_intermediate_size 2560, norm_topk_prob true, n_group 8, topk_group 4, routed_scaling_factor 2.5. No shared-expert field. mlp_layer_types is three dense followed by 49 sparse. layer_types is 43 sliding_attention and 9 full_attention, full attention at indices 5, 11, 17, 23, 29, 35, 41, 47, 51. sliding_window 512. rope_parameters: sliding_attention rope_type default, rope_theta 10000; full_attention null. No rope_scaling key. final_logit_softcapping 30.