Naive-N0.5-Flash

ADDED 2026-10-18. Open-weight MoE continued from Xiaomi MiMo-V2.5, weights public 2026-09-27 (HF API createdAt 2026-09-27T14:14:32Z, lastModified the same day). The catalog had models/mimo-v2-6-flash-rl, which shares the 309B / 48-layer / 256-expert shape, but not this continued-pretrain. RUNNABLE WITH EXPERT OFFLOAD: 309B total but the card activates 15.5B per token and routes only 8 of 256 experts, with no shared expert, so the per-token expert traffic is a small slice of the checkpoint and the expert bank can sit in system RAM while attention stays resident. Same routing width as models/mimo-v2-6-flash-rl. PARAMS: params_b 309 and active_params_b 15.5 are the card nameplate, not a count derived here. HF API usedStorage is 617,823,283,433 bytes across 49 shards, about 575 GiB, which matches a BF16 dump of a 309B model (309e9 x 2 = 618 GB) and is why this only makes sense with expert offload rather than as a dense-in-VRAM model. The card's deployment section separately says the FP8 weights occupy about 315 GB. flops_per_token_active = 2 x 15.5e9 = 3.1e10 is the schema one-pass approximation off the vendor active count. GEOMETRY from config.json fetched 2026-10-18 (commit 9c9a11e8d2a6): model_type naive_n05_flash, custom code via auto_map, hidden 4096, 48 layers, 64 query heads, 4 kv heads, head_dim 192, v_head_dim 128, so gqa_ratio 16 on the sparse layers. MoE: n_routed_experts 256, num_experts_per_tok 8, n_shared_experts null so shared_expert is false, moe_intermediate_size 2048. moe_layer_freq is 0 followed by 47 ones, so only layer 0 is a dense FFN (intermediate_size 16384) and the other 47 are MoE. hybrid_layer_pattern has 9 zeros and 39 ones. The card says the zeros are DeepSeek Sparse Attention and the ones are sliding-window attention, and the pattern puts sparse attention at indices 0, 5, 11, 17, 23, 29, 35, 41, 47. Sliding window 128, and the sliding layers use 8 kv heads (swa_num_key_value_heads 8) at the same head dims. DSA indexer: index_top_k 2048, index_n_heads 16, index_head_dim 128, indexer stored in fp8 e4m3. There is no full-attention layer and no kv_lora_rank, so this is GQA, not MLA. silu, vocab 152576, tie_word_embeddings false, rope_theta 10000000, partial_rotary_factor 0.334, rope_scaling type default, max_position_embeddings 1048576, so context_scaling_method is native. KV CACHE. The card publishes no per-token cache size. The card also says the full KV cache is retained even on the sparse layers (the indexer selects 2048 tokens but does not drop the cache), so all 9 sparse layers grow with context. Derived, bf16, K at head_dim 192 and V at v_head_dim 128: 9 x 4 kv heads x (192 + 128) x 2 bytes = 23,040 B/token. The 39 sliding layers hold a fixed 128-token window at 8 kv heads: 39 x 8 x (192 + 128) x 2 x 128 = 25,559,040 bytes, about 24.4 MiB, which does not grow with context and is not in kv_cache_bytes_per_token. The indexer is a separate per-token state (index_head_dim 128, fp8) and is excluded, the same treatment the catalog gives the GLM-5.3-Flash indexer. If every layer cached the full context at the sparse-layer geometry the figure would be 48 x 4 x 320 x 2 = 122,880 B/token, about 5x the recorded number. ENGINES AND QUANTS. The card's quick start is Hugging Face Transformers >=5.17.0 with trust_remote_code, FP8 mixed precision, on FP8-capable NVIDIA GPUs, and it recommends temperature 1.0 and top_p 0.95. The acknowledgments name SGLang as the inference stack the lab built on, but the card does not publish an SGLang or vLLM flag, and it does not say either engine can keep attention resident while offloading experts. That split is the architecture (8 of 256 routed, attention and the layer-0 dense MLP are not in the expert bank), not a measured engine flag. The model tree lists four community quantizations tagged for llama.cpp, LM Studio, Jan and Ollama; the vendor repo itself ships no GGUF and no measured single-GPU offload recipe. KTransformers is not named. BENCHMARKS ARE VENDOR CLAIMS. The README prints no numbers; they are read from assets/coding-benchmarks.pdf on the same repo, and the alt text on the lab's research page (https://naive.ai/en/research/, fetched 2026-10-18) repeats them: NL2Repo-Bench 71.9, DeepSWE v1.1 67.8, Terminal-Bench 2.1 86.7, SWE-bench Pro 73.6, ProgramBench Almost@1 17.5, Agents' Last Exam (ALE-CLI) 32.4, FrontierSWE v1 78.2. The card says its own runs used Claude Code 2.1.207, a 1M context, temperature 1.0, top_p 0.95, and only file I/O plus Bash. Not re-run here. The figure does not print SWE-bench Verified, GPQA Diamond, AIME or MMLU-Pro, so those are not claimed. BEATS models/deepseek-v4-1-flash-1017, the catalog's only record with a published NL2Repo-Bench number, on NL2Repo-Bench: 71.9 vs 68.2. Both numbers are vendor claims and were not re-run on one harness. Same offload class: both are large-total MoE with a far smaller active footprint and a handful of experts routed per token, so the expert bank can sit in RAM. This one activates 15.5B of 309B and routes 8 of 256; DeepSeek-V4.1-Flash-1017 activates 8B of 552B on prefill (16B on decode) and routes 6 of 384. The same figure's DeepSWE v1.1 score of 67.8 does NOT beat that record (77.0), Terminal-Bench 2.1 at 86.7 does NOT beat models/mimo-v2-6-flash-rl (87.6) or models/mimo-v2-6-pro-rl (89.9), and SWE-bench Pro at 73.6 is not compared because the catalog's highest published SWE-bench Pro is models/ling-3-1-flash at 68.74 on a different harness (LingCode) and the Naive figure does not name its SWE-bench Pro harness. NL2Repo-Bench is the qualifying comparison. The figure also prints DeepSeek-V4.1-Flash at 64.0 on NL2Repo, which is the parent's number; the catalog record is the 1017 update at 68.2, and 71.9 still beats that.