Qwen3.5-397B-A17B

ADDED 2026-10-18. Open-weight vision-language MoE, the largest Qwen3.5 checkpoint, released 2026-02-16 (HF API createdAt). RUNNABLE WITH EXPERT OFFLOAD: the checkpoint is 403.398B parameters but the card activates 17B per token and routes only 10 of 512 experts plus one shared expert, so the per-token expert traffic is a small slice of the BF16 checkpoint and the expert bank can sit in system RAM. PARAMS: HF safetensors metadata total is 403,397,928,944, of which 403,397,920,304 are BF16 and 8,640 are F32, so params_b is 403.398. The card rounds the language model to 397B total and 17B activated; the extra is the vision tower plus the MTP block, the same split the smaller Qwen3.5 cards describe, and it is not subtracted here because the per-tensor split was not counted. active_params_b is the card 17B figure, not a count derived here. flops_per_token_active = 2 x 17e9 = 3.4e10 is the schema one-pass approximation. BEATS models/qwen3-5-122b-a10b, the catalog best open-weight record with a published MMLU-Pro number, on MMLU-Pro: 87.8 vs 86.7. Both numbers are vendor claims and were not re-run on one harness here. Same offload class: both are Qwen3.5 hybrid MoEs routing a small fixed set of experts (10 of 512 here, 8 of 256 there) with an active footprint far below the total. It does NOT beat models/kimi-k2-6 on LiveCodeBench v6 (83.6 vs 89.6) or GPQA Diamond (88.4 vs 90.5); the Kimi card publishes no MMLU-Pro number, so that comparison is not made. Its SWE-bench Verified 76.4 is above models/qwen3-6-35b-a3b (76.0) but below models/mimo-v2-6-flash-rl (79.6) and models/kimi-k2-6 (80.2), so MMLU-Pro against models/qwen3-5-122b-a10b is the qualifying comparison. AIME26 is 91.3, below the catalog AIME 2026 leaders, and is not used. GEOMETRY from config.json fetched 2026-10-18 (commit 8472618112ab): top-level model_type qwen3_5_moe, text_config model_type qwen3_5_moe_text, hidden 4096, 60 layers, 32 query heads, 2 kv heads, head_dim 256, so gqa_ratio 16. MoE: 512 experts, 10 per token, moe_intermediate_size 1024, shared_expert_intermediate_size 1024 so shared_expert is true. silu, vocab 248320. Attention is hybrid: layer_types is 45 linear_attention and 15 full_attention, full_attention_interval 4. Linear block: linear_num_key_heads 16, linear_num_value_heads 64, key and value head dim 128. KV (bf16, full-attention layers only): 15 x 2 kv heads x 256 x 2 (K and V) x 2 bytes = 30,720 B/token. The 45 linear-attention layers hold a fixed recurrent state, not a growing cache, so they are not in kv_cache_bytes_per_token. RoPE is native mrope: rope_type default, rope_theta 10000000, partial_rotary_factor 0.25, mrope_interleaved true, mrope_section [11, 11, 10], no rope_scaling key. License Apache-2.0. ENGINES: the card says the weights are compatible with Hugging Face Transformers, vLLM, SGLang and KTransformers. QUANT: the checkpoint is BF16 (no quantization_config; safetensors are 403,397,920,304 BF16 plus 8,640 F32), so a Q4 expert bank is a quantisation the user applies, not one the repo ships. Expert offload while attention stays resident is the architecture: 10 of 512 routed experts plus one shared expert of width 1024, with the 15 full-attention layers and the 45 linear-attention recurrent states outside the expert bank. The card does not publish a measured single-GPU offload recipe, so that split is not recorded as a measured flag.