Qwen3-30B-A3B Hugging Face config.json (raw)

Read 2026-10-04 for the FLOP-crossover record. hidden_size 2048, 48 layers, 32 attention heads, 4 KV heads, head_dim 128 (so n_q*d_h = 4096 = 2x hidden_size, because head_dim is set independently of hidden_size/num_heads), intermediate_size 6144 for dense layers, moe_intermediate_size 768, num_experts 128, num_experts_per_tok 8, vocab 151936, max_position_embeddings 40960. This is the row that makes the crossover table non-trivial: a 30B model with only 3B active parameters has a MUCH lower L* than any dense 30B because the per-token GEMM is small while the attention term is unchanged.