NVIDIA Nemotron 3.5 Super VL 120B-A12B

ADDED 2026-10-18. Open-weight vision-language hybrid MoE. HF API createdAt 2026-10-12 for nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16; the card's Release Date line says 2026-10-15. This record is the BF16 post-trained checkpoint, not the NVFP4 quant and not the LiLiCorr drafter. Parent of models/nemotron-3-5-bolt-74b-a9b, which the Bolt card says was compressed from this checkpoint with Star Elastic. RUNNABLE WITH EXPERT OFFLOAD, with a caveat: 120B total and the card's 12B active, but config.json sets num_experts_per_tok to 22 of 512 routed experts plus one shared expert (n_shared_experts 1). That is more than twice the routed width of models/qwen3-5-122b-a10b (8 of 256) and of the catalog's Bolt child (9 routed). Expert offload still works because only those 22 experts move per token and the other ~490 stay in RAM, but the per-token expert traffic is the wide end of the small-active class, not the Flash-Next shape. The card and the technical report both state 12B active; neither states the routed count, so experts_per_token is the config value and the 12B figure is the vendor active count, not a count derived here. BEATS models/qwen3-5-122b-a10b, the catalog's ~10-12B-active open-weight MoE with a published GPQA Diamond number, on three benchmarks in NVIDIA's own Table 10 (technical report, OpenCode harness for SWE, Terminus 2 with a 3 hour timeout for Terminal-Bench 2.1): GPQA Diamond 89.52 vs the Qwen card's 86.6, Terminal-Bench 2.1 60.67 vs that card's published row being absent so the comparison on Terminal-Bench is against the report's own Qwen3.5 122B-A10B column of 48.70, and SWE-bench Pro 51.78 vs the report's Qwen3.5 column of 45.60. The GPQA comparison mixes NVIDIA's harness with the Qwen card's vendor claim and was not re-run here; the Terminal-Bench and SWE-bench Pro comparisons are inside one NVIDIA table. It does NOT beat models/qwen3-8-flash-next on the report's own rows (Terminal-Bench 2.1 60.67 vs 85.80, SWE-bench Verified 71.10 vs 72.60, GPQA Diamond 89.52 vs 92.68), and it does not beat the report's GLM-5.3 Flash column on SWE-bench Verified (71.10 vs 79.10) or Terminal-Bench 2.1 (60.67 vs 82.70). The qualifying win is the ~10-12B-active class, against models/qwen3-5-122b-a10b, not the large-total sparse class. The same table's SWE-bench Verified 71.10 is below the 72.2 vendor claim already on models/qwen3-5-122b-a10b, so SWE-bench Verified is not the win. GEOMETRY from config.json fetched 2026-10-18: top-level model_type nemotron_h_omni, architectures NemotronH_Omni_Reasoning_V3, llm_config model_type nemotron_h. hidden 4096, 88 blocks = 40 mamba + 40 moe + 8 attention at indices 7, 16, 25, 36, 47, 58, 69, 78. Attention heads 32 / kv heads 2 / head_dim 128 so gqa_ratio 16. vocab 131072, max_position_embeddings 1048576, sliding_window null, no rope_scaling key, so context_scaling_method is native. Unlike the Bolt child, this config is uniform: n_routed_experts 512, num_experts_per_tok 22 and moe_intermediate_size 2688 are scalars, not per-block lists. Shared expert: n_shared_experts 1, moe_shared_expert_intermediate_size 5376, moe_latent_size 1024. Mamba-2: mamba_num_heads 128, mamba_head_dim 64, ssm_state_size 128, conv_kernel 4, mamba_hidden_act silu, mlp_hidden_act relu2. One MTP layer (num_nextn_predict_layers 1, mtp_layers_block_type attention+moe). Vision: vision_config model_type radio, hidden 1280, 32 layers, 16 heads, patch 16, image 224; top-level vit_hidden_size 1280. KV (bf16, attention blocks only): 8 layers x 2 kv heads x 128 x 2 (K and V) x 2 bytes = 8192 B/token. The Mamba recurrent state is the other memory term and is not in that number. flops_per_token_active = 2 x 12e9 = 2.4e10 is the schema approximation from the vendor's 12B-active figure; hybrid Mamba plus the shared expert mean it is not exact. License OpenMDW-1.1. Engines named on the card: vLLM (cookbook, including the NVFP4 checkpoint), TensorRT-LLM, SGLang, llama.cpp (community GGUF, not the BF16 weights), and Transformers via the repo's custom modeling code.