MiniCPM-V-4_6

CONTEXT SCALING: null is a VERIFIED value here, checked against config rather than left blank. config.json (openbmb/MiniCPM-V-4_6) declares rope_scaling null, max_position_embeddings 262144. Verified null. THE MOST ARCHITECTURALLY INTERESTING RECORD IN THIS SLICE, and the only non-dense one. TWO-PART MODEL. (1) Vision: SigLIP, 27 layers, hidden 1152, 16 heads, patch 14, image_size 980, gelu_pytorch_tanh, insert_layer_id 6 -- vision features are injected at layer 6 of the LLM rather than only at the input. image_size 1120 at top level, image_token_id 248056, video_token_id 248057. (2) LLM: text_config model_type qwen3_5_text -- 24 layers, hidden 1024, 8 heads / 2 KV heads, head_dim 256 (4x wider than hidden_size/num_heads), intermediate 3584, vocab 248,094, max_position 262,144, rope_theta 1e7, partial_rotary_factor 0.25 (only a quarter of head dims get RoPE), tie_word_embeddings true, attn_output_gate true, mtp_num_hidden_layers 1 (multi-token prediction head), linear_conv_kernel_dim 4, mamba_ssm_dtype float32. ARCHITECTURE = hybrid-attention-ssm, and this is the whole point: the layer_types array spells out exactly 6 full_attention layers at indices 3, 7, 11, 15, 19, 23 and 18 linear_attention layers everywhere else, in a strict 3:1 repeating pattern. KV CACHE: 6 full-attention layers * 2 kv_heads * 256 head_dim * 2 * 2 bytes = 12,288 B/token (12 KiB). The 18 linear_attention layers hold a FIXED-SIZE recurrent state (linear_key_head_dim 128, linear_value_head_dim 128, linear_num_key_heads 16, linear_num_value_heads 16) that does not grow with context, so they contribute a constant ~18 * 16 * 128 * 2 * 4 bytes ~= 0.3 MiB per sequence TOTAL, independent of context length. THIS IS 12x SMALLER PER TOKEN than MiniCPM-V-4_5 (147,456 B) despite being a similar generation, and it is the only record here where a 262,144-token context is a realistic serving target: 262,144 * 12,288 B = 3.0 GiB resident KV. The 18:6 ratio is the knob to tune -- every layer converted from full to linear attention removes 2,048 B/token. VISION TOKENS: not derivable from this config -- the resampler/tiling fields (query_num, slice_config) that MiniCPM-V-4_5 exposes are absent here, and the vision_config block carries no image_seq_len. Given the shared SigLIP-980/patch-14 tower the per-slice cost is likely comparable to the 4.5's 64 tokens, but that is an INFERENCE and is deliberately not asserted here. params_b 1.300 = HF API safetensors total (1,300,428,016) -- a 1.3B multimodal model, so this generation is a large step down in size from 4_5's 8.7B. transformers_version 5.7.0. Released 2026-04-13, the most recent checkpoint in this slice. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=hybrid-attention-ssm. REQUIRED KERNEL CLASS: two kernel families in one graph - a chunked linear-attention / delta-rule kernel for the SSM or linear-attention layers, and an ordinary or MLA decode kernel for the full-attention minority - plus a fixed-size recurrent state allocator. Three independent records say the recurrent branch is the hard part: flop/ssm-recurrent-state-update (the state is a fixed per-sequence tax that never amortises, so it scales with concurrency and NOT with context); flop/ssm-sequential-scan-tensor-core-mismatch (the recurrence is a multiplicative chain that tensor cores cannot express, so it must run as a sequential scan; the published SSD-vs-fused-scan gap is 2-8x for identical mathematics); and llama.cpp's own dispatch table, which puts every hybrid and SSM architecture in its supports_sm_tensor == false list [[mdl2-llamacpp-arch-class-registry]] - the engine agreeing with the arithmetic. That same list also shows llama.cpp needed ONE shared hybrid code path (llm_arch_is_hybrid, 18 entries: JAMBA, FALCON_H1, PLAMO2, GRANITE_HYBRID, LFM2, LFM2MOE, NEMOTRON_H, NEMOTRON_H_MOE, QWEN3NEXT, KIMI_LINEAR, BAILINGMOE3, KIMI_K3, GLM5_NEXT, QWEN35, QWEN35MOE, QWEN4EXP, DEEPSEEK4, MINIMAX_01) covering exactly the families vLLM and SGLang each ship as separate rows, which is why hybrid support arrives per engine release rather than at once. SUPPORTED BY (cited): vLLM native rows MambaForCausalLM, Mamba2ForCausalLM, JambaForCausalLM, FalconMambaForCausalLM, FalconH1ForCausalLM, NemotronHForCausalLM, GraniteMoeHybridForCausalLM, Qwen3NextForCausalLM, KimiLinearForCausalLM, Lfm2ForCausalLM, Lfm2MoeForCausalLM, OlmoHybridForCausalLM, KimiK3ForConditionalGeneration, BailingMoeV3ForCausalLM, Qwen3_5 rows [[mdl2-vllm-supported-models-2026-10]]; SGLang generative table names Granite 4.0 micro as 'hybrid Mamba-MoE ... combining gated short convolutions with a small number of grouped query attention blocks', Kimi Linear as 'hybrid linear attention model ... Kimi Delta Attention (KDA) for up to 6x faster decoding and 75% KV cache reduction vs full attention', Nemotron Nano as 'hybrid Mamba-Transformer architecture', Ling as 'hybrid MoE model: per-layer mix of DeepSeek-style MLA full attention and Qwen3 Gated-Delta-Net (GDN) linear attention', LFM2 as 'hybrid backbone', Mamba-Codestral as 'Mamba2 ... holds no KV cache', and its multimodal page carries the descriptor 'hybrid KDA/MLA' [[mdl2-sglang-generative-models]] [[mdl2-sglang-multimodal-language-models]]. TensorRT-LLM has rows for NemotronHForCausalLM, Qwen3NextForCausalLM, Qwen3_5MoeForCausalLM, Qwen4ExpForCausalLM, KimiLinearForCausalLM and KimiK3ForConditionalGeneration but NO row at all for MambaForCausalLM, Mamba2ForCausalLM, FalconH1ForCausalLM, Lfm2ForCausalLM or GraniteMoeHybridForCausalLM [[mdl2-trtllm-supported-models]]. PER-FAMILY FEATURE TRAPS, all from the TRT-LLM key-models matrix, which is the only per-architecture feature matrix in this repo [[mdl2-trtllm-supported-models]]: Qwen3NextForCausalLM has Sliding Window Attention = No and Speculative Decoding = No; Kimi K3 has no MTP or EAGLE-3 head at all (spec method DSpark only) and needs Blackwell SM100; MiniMax-M3 sparse-attention path supports neither KV cache reuse nor MTP in this release; DeepSeek-V4 and Step-3.7 are 'Untested' for disaggregated serving. KERNEL BUGS THAT ARE ARCHITECTURE-SPECIFIC AND REPRODUCED: (1) vLLM's EAGLE last-block drop (drop_eagle_block) costs 6-10 points of prefix-cache hit rate, and on a hybrid GDN/Mamba layout the scheduler alignment unit is 1,648 tokens instead of a 16-token dense block, so the same protection costs ~100x more cache hit - 97.5% to 86.9-91.7% hit rate and 30-40% batch throughput on prefix-reusing workloads [[eagle-prefix-cache-last-block-drop]]. (2) llama.cpp decode collapses on hybrid gated-deltanet GGUF models at high KV position: 33 t/s at 68K to 1.4 t/s at 91K with the model fully resident, across KV quant types and with FA on or off, reproduced on Windows and WSL2 on master and release [[gated-deltanet-decode-collapse-long-context]]. (3) llama.cpp's SYCL path has a regression specific to hybrid linear-attention graphs - qwen3next/qwen3_35 architecture models crash in ggml_sycl_op_mul_mat or return empty output on 2x Arc Pro B60, bracketed between builds b8477 (works) and b9479 (fails), while dense-attention models on the same cards are unaffected [[sycl-hybrid-linear-attention-arch-crash-intel-arc]]. (4) FlashInfer rejects large head sizes at engine init for the head geometry these hybrids introduce [[flashinfer-rejects-large-head-dim]]. Two kernel classes plus a vision tower: the 18 linear-attention layers need the GDN path, the 6 full-attention layers need ordinary GQA attention, and the SigLIP-980/patch-14 tower needs its own forward. TRT-LLM has MiniCPMV4_6ForConditionalGeneration with the footnote 'Requires transformers>=5.7.0: MiniCPM-V 4.6 was upstreamed into transformers as a native model type (minicpmv4_6) and the checkpoint ships no remote code (auto_map) to fall back on', and that the Qwen3.5-hybrid text tower runs in BF16 while image/video/text inputs are supported in this release [[mdl2-trtllm-supported-models]]. vLLM's registry has a MiniCPMV row covering openbmb/MiniCPM-V-4_5 and 4_6 [[mdl2-vllm-supported-models-2026-10]]. llama.cpp's supports_sm_tensor == false list includes MINICPM3 (a different checkpoint) but NOT a minicpmv4_6 entry in llm_arch_is_hybrid, so on that engine's own registry the 4.6 hybrid layout is not in the hybrid class [[mdl2-llamacpp-arch-class-registry]]. TRT-LLM's multimodal matrix marks this row Untested for CUDA graph, attention DP and chunked prefill [[mdl2-trtllm-supported-models]]. This checkpoint's own hybrid linear-attention layer shape is the one llama.cpp's SYCL regression bisects on Arc, by architecture family [[sycl-hybrid-linear-attention-arch-crash-intel-arc]]. CONTEXT_SCALING_METHOD (native): SCHEME RECLASSIFIED (2026-10-04): was null. rope_parameters rope_type 'default' at theta 1e7, max_position_embeddings 262144. LAYER LAYOUT (minicpm-v-4-6): layer_types read verbatim from config, one entry per layer; attention_layer_indices are the 0-based positions of the full-attention entries.