stepfun-ai/Step-5-Preview config.json

Fetched 2026-10-18 from resolve/main/config.json. Top-level model_type step5_vl, architectures Step5ForConditionalGeneration, with a perception-encoder vision tower (image_size 728, width 1536, 47 layers) and a text_config. text_config: model_type step5, architectures Step5ForCausalLM, hidden_size 4096, intermediate_size 13824, num_hidden_layers 92, num_attention_heads 64, num_attention_groups 4, head_dim 192 for both full and sliding attention, hidden_act silu, vocab_size 128896, max_position_embeddings 1048576, sliding_window 512. MoE: use_moe true, moe_num_experts 352, moe_top_k 8, num_experts_per_tok 8, moe_intermediate_size 1536, share_expert_dim 1536, moe_router_scaling_factor 3.0, norm_expert_weight true. moe_layer_list covers layers 3 through 90 (88 MoE layers); layers 0, 1, 2 and 91 are dense. layer_types lists 95 entries because num_nextn_predict_layers is 3: the 92-layer backbone has 23 full_attention and 69 sliding_attention, with full attention at indices 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63, 67, 71, 75, 79, 83, 87, 91. sparse_config enables a sparse indexer (topk 512, proxy_dim 256) applied only to full_attention layers. rope_scaling type llama3, factor 1.0, original_max_position_embeddings 1048576, high_freq_factor 32, low_freq_factor 1.0. quantization_config quant_method fp8, fmt e4m3, weight_block_size [128, 128].