Step 5 Preview

ADDED 2026-10-18. Open weights on Hugging Face since 2026-09-20 (HF API createdAt 2026-09-20T03:15:06Z); the vendor model page said the weights would land on 2026-10-15, and the checkpoint was public when checked on 2026-10-18. Card nameplate: sparse Mixture-of-Experts, 600B total, 27B active per token, up to 1M context, text and image in, text out. config.json agrees on the routing: text_config model_type step5, hidden_size 4096, num_hidden_layers 92, num_attention_heads 64, num_attention_groups 4 so 4 KV heads, head_dim 192, moe_num_experts 352, num_experts_per_tok 8, moe_intermediate_size 1536, share_expert_dim 1536, vocab_size 128896, max_position_embeddings 1048576, sliding_window 512. moe_layer_list is layers 3 through 90, so 88 MoE layers and 4 dense layers (0, 1, 2, 91). A shared expert is present because share_expert_dim is 1536, a positive width, not an inference from a missing key. params_b 600 and active_params_b 27 are the vendor nameplate figures; the safetensors shards were not summed here. flops_per_token_active = 2 x 27e9 = 5.4e10 is the schema one-pass approximation off that vendor figure. ATTENTION SPLIT. layer_types lists 95 entries because num_nextn_predict_layers is 3 (a speculative head, not part of the decoded backbone). The 92-layer backbone is 23 full_attention and 69 sliding_attention. Full attention sits every fourth layer at indices 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63, 67, 71, 75, 79, 83, 87, 91. sparse_config turns on a sparse indexer (topk 512, proxy_dim 256) for the full_attention layers only. KV CACHE. The card publishes no per-token cache size, so this is derived and counts only the 23 full-attention layers, the same treatment the catalog gives hybrid sliding-window models. 23 x 2 (K and V) x 4 KV heads x head_dim 192 x 2 bytes (bf16) = 70,656 B/token. The 69 sliding layers hold a fixed window of 512 and do not grow with context: 69 x 2 x 4 x 192 x 2 x 512 = 108,527,616 bytes, about 103.5 MiB, which is not part of the per-token figure. If every backbone layer cached globally the figure would be 92 x 2 x 4 x 192 x 2 = 282,624 B/token, four times higher. context_scaling records the llama3 rope_scaling block (factor 1.0, original_max_position_embeddings 1048576, high_freq_factor 32); context_scaling_method is llama3. The 1M window is what config declares, not an extension beyond it. OFFLOAD SHAPE. 8 of 352 experts routed per token plus one shared expert of width 1536, so the expert bank can sit in system RAM while the attention stack, the 4 dense MLPs, the router and the shared expert stay resident. At Q4 the active footprint is about 27e9 x 0.5 = 13.5 GB of weights read per token, plus the fixed 103.5 MiB sliding-window state. The BF16 checkpoint is 600B parameters, about 1.2 TB, which is why this only makes sense with expert offload rather than as a dense-in-VRAM model. 8 routed experts is the same routing width as models/beam-1-0-501b-a23b and models/glm-5-3-flash, so the bytes moved per token are in the same class, not the datacenter-only case of routing dozens of experts. 27B active is well above the 6B-active Qwen3.8-Flash-Next reference shape, so decode under expert offload is several times more expensive than the small-active class. ENGINES. The model card names vLLM and SGLang for serving and Transformers for offline generation, and ships an inference/ directory in the repo. The published configs start at FP8 on 4 B300 GPUs or 8 H200 GPUs and go up to BF16 on 16 H200 GPUs; none of them is a single-GPU expert-offload recipe, and the card does not say whether vLLM or SGLang can keep attention resident while offloading experts. That split is the architecture, not a measured engine flag. llama.cpp and GGUF are not named. The checkpoint on the card is FP8 (quant_method fp8, e4m3, block size 128), and a separate Step-5-Preview-BF16 repo exists. BENCHMARKS ARE VENDOR CLAIMS from the StepFun model page, which the Hugging Face card points to for methodology, at the High reasoning setting. Not re-run here. BEATS models/beam-1-0-501b-a23b, the catalog's highest recorded GPQA Diamond number, on GPQA Diamond: 93.5 vs 90.6. Same class: both are large-total MoE with a much smaller active footprint and 8 experts routed per token, so the expert bank can be offloaded to RAM. Step 5 Preview activates 27B of 600B with 8 of 352 experts; Beam activates 23.4B of 501.3B with 8 of 256. It also beats models/mimo-v2-6-flash-rl (90.1) and models/qwen3-8-27b (89.2) on that benchmark. On Terminal-Bench v2.1 the same table gives 85.0, which beats models/beam-1-0-501b-a23b (80.1) and models/qwen3-8-27b (73.0) but does NOT beat models/mimo-v2-6-flash-rl (87.6), so Terminal-Bench is not the qualifying comparison. The page's DeepSWE v1.1 score of 67.7 is not SWE-bench Verified or SWE-bench Pro and is not used as the comparison.