K2-Horizon-MoVA-36B-A4B

ADDED 2026-10-18. Open-weight sparse MoE in the ~4B-active class, with Mixture-of-Values attention on top of ordinary GQA. RUNNABLE WITH EXPERT OFFLOAD: 37.445B stored but the card nameplate is 36B total / 4B activated, and only 8 of 100 routed experts fire per token, so the per-token expert traffic is a small slice of the checkpoint. The first three layers are dense (mlp_only_layers 0,1,2, intermediate_size 6144) and the remaining 45 are MoE. PARAMS: HF API safetensors total is 37,444,792,020 BF16, so params_b is 37.445 and the card 36B is the rounded nameplate. active_params_b is the card 4B figure, not a count derived from the config: the shared expert, the three dense layers and the MoVA value experts are not given as a parameter sum. flops_per_token_active = 2 x 4e9 = 8.0e9 is the schema one-pass approximation off that vendor figure. BEATS models/xing4-0-29b-a4b, the catalog best ~4B-active record with a published Terminal-Bench 2.1 number, on Terminal-Bench 2.1: 58.6 vs 57.50. Both numbers are vendor claims and were not re-run on one harness. The K2 figure is the card benchmark table, which says baseline scores are from Artificial Analysis and its own row was run at high reasoning effort (temperature 1.0, top_p 0.95); the Xing figure is the Xing card own table under the terminus-2 harness. Same offload class: both are ~30-37B total MoE with 4B active and a handful of experts routed per token, so the expert bank can sit in RAM. This one routes 8 of 100; Xing routes 4 of 64. The same table puts K2 ahead of the card Qwen3.6-35B-A3B row on Terminal-Bench 2.1 (58.6 vs 44.9), but that row is the card own number for Qwen and is lower than the 51.50 the Xing card reports for the same model, so the catalog comparison stays Xing. It does NOT beat models/ornith-1-5-35b-a3b on GPQA Diamond (80.8 vs 88.6), and the card prints no SWE-bench Verified, AIME or MMLU-Pro number, so Terminal-Bench 2.1 is the qualifying comparison. The table also lists tau3-Banking 26.8, SciCode 38.9, HLE without tools 25.2, CritPt 2.1, AA-LCR 66.3 and AA-Omniscience accuracy 18.8, none of which are the catalog qualifying benchmarks. ARCHITECTURE from config.json fetched 2026-10-18: model_type k2_horizon, architectures K2HorizonForCausalLM, custom code via auto_map (configuration_k2_horizon.py, modeling_k2_horizon.py), so it does not load in stock transformers without the repo code. hidden 2560, 48 layers, 32 query heads, 8 kv heads, head_dim 128 explicit, so gqa_ratio 4. MoE: 100 routed experts, 8 per token, 1 shared expert, moe_intermediate_size 768, router_score_func sigmoid, router_scaling_factor 2.5, norm_topk_prob true, moe_gate_bias true. MoVA is a second, smaller routing: mova_num_experts 64, mova_num_experts_per_tok 4, attention_gate_func softplus. silu, vocab 250624, tie_word_embeddings false, bos 0, eos 1. KV (bf16): 48 layers x 8 kv heads x 128 head_dim x 2 (K+V) x 2 bytes = 196,608 B/token. The MoVA value routing is not in that figure; the config does not say the value experts shrink the cached KV, so the standard GQA formula is what is recorded. CONTEXT: max_position_embeddings 524288. rope_parameters is rope_type default with rope_theta 1e7 and no scaling factor, matching the card native 512K window trained in from midtraining stage 3, so context_scaling_method is native. ENGINES AND LICENSE. The card quick start is vLLM (tensor-parallel 2, expert-parallel, reasoning-parser k2_horizon) and an SGLang recipe validated on 2x H200, plus Transformers 5.15.0 with trust_remote_code. It does not publish a measured single-GPU offload recipe; the offload case is the architecture (8 of 100 routed, 4B active of 37.4B stored), not a measured engine result. License Apache-2.0 (card license field and HF tag license:apache-2.0). Weights repo created 2026-09-01 (HF API createdAt 2026-09-01T23:01:25Z). Training data and code are public (TxT360-v2, github.com/ifm-ai/xllm); the technical report was still listed In Progress as of the card last-updated date 2026-09-28. BENCHMARKS ARE VENDOR CLAIMS, added 2026-10-18 from the same card the geometry above cites. BEATS models/xing4-0-29b-a4b, the catalog best ~4B-active record with a published Terminal-Bench 2.1 number, on Terminal-Bench 2.1: 58.6 vs 57.5. Both numbers are vendor claims and were not re-run on one harness. Same offload class: both are ~30-37B total MoE with 4B active and a handful of experts routed per token (this one routes 8 of 100, Xing routes 4 of 64 plus one shared expert), so the expert bank can sit in RAM while attention stays small. The same table reports GPQA Diamond 80.8, which does NOT beat models/qwen3-6-35b-a3b (86.0), so that is not the qualifying comparison. The card says baseline rows are from Artificial Analysis and its own row is at high reasoning effort.