DeepSeek-V4-Flash

per-layer compress_ratios (config), length 43 = num_hidden_layers: layers 0-1 ratio 0 (uncompressed), layer 2 ratio 0, then layers 3-41 ALTERNATE ratio 4 (CSA, compressed-sparse) and ratio 128 (HCA baseline), layer 42 ratio 0. Paired with sliding_window 128, num_hash_layers 3 and index_topk 512. The 4-vs-128 split is a 32x per-layer difference, so no single average ratio describes this model.

KV GEOMETRY: kv_cache_bytes_per_token REMAINS NULL, and the reason is now precise rather than vague. The trap on this family is assuming MLA: config has NO kv_lora_rank, so the DeepSeek-V2/V3 latent formula (layers x (kv_lora_rank + qk_rope_head_dim) x 2) does not apply either. V4 instead declares a per-layer compress_ratios array of ints, which transformers maps to attention layer types via _COMPRESS_RATIO_TO_LAYER_TYPE: 0 -> sliding_attention (full 128-token sliding-window KV), 4 -> compressed_sparse_attention (CSA, compress_rate 4, plus a Lightning Indexer running its own compressor at index_head_dim 128 over the same windows), 128 -> heavily_compressed_attention (HCA, compress_rate 128, one compressed entry per 128 tokens). Flash: 3 sliding / 21 CSA / 20 HCA over 43 layers. Pro: 1 / 30 / 31 over 61. Why this is not derivable to a single number: every compressor layer retains its sliding-window KV PLUS a growing compressed-KV list PLUS (on CSA layers) a second indexer series plus a 2x-compress_rate_csa overlap buffer, and the three interleave. An exact figure needs the CSA/HCA overlap accounting in modeling_deepseek_v4.py, not config arithmetic. What IS safe to state: the naive GQA reading (num_layers x num_kv_heads 1 x head_dim 512 x 2 x 2 = 88,064 B/token, 86.0 KiB) is wrong because it charges every layer a full head_dim 512 K and V per token, while HCA layers charge head_dim x 2 x 2 / 128 per token - about 16.0 B/token/layer, a 128x reduction on 20 of the layers. So the naive figure OVERSTATES V4's KV, and the direction matters for serving: it would say V4-Flash needs 86.0 KiB/token when its compressed design is precisely what makes 1M context tractable. The biggest serving-economics jump in this slice, and the one field I could NOT fully derive. Vendor: 284B total / 13B activated, 1M context (max_position_embeddings 1048576 via YaRN factor 16 over an original 65536 - a much smaller extension factor than V3's 40). kv_cache_bytes_per_token is null because kv_lora_rank is ABSENT from the V4 config. V4 replaces the plain MLA latent with a compressed/quantized cache: config has expert_dtype fp4, hc_mult 4, num_hash_layers 3, o_lora_rank 1024, a per-layer compress_ratios array that is mostly 4 (interleaved with 128 every 5th layer), compress_rope_theta 160000, and sliding_window 128. A reader should NOT assume the V3 latent formula carries over; the geometry is genuinely different and needs the V4 modeling code to state precisely. Architecture is otherwise DeepSeek-V3-like: 43 layers (not 61), hidden 4096, 256 routed + 1 shared expert, 6 per token (not 8), moe_intermediate_size 2048, q_lora_rank 1024, head_dim 512 with qk_rope_head_dim 64, num_key_value_heads 1. Weights ship pre-quantized (fp8 e4m3, 128x128 block) so the bf16 total is smaller than a naive 284B suggests. MIT. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=mla. REQUIRED KERNEL CLASS: an MLA decode kernel. MLA is not a variant of the GQA kernel - the cache is a shared low-rank latent (d_c) plus a decoupled RoPE key, not per-head K/V, so an engine that lacks a dedicated MLA decode path cannot serve the class at all. SUPPORTED BY (cited): vLLM native rows DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, DeepseekV32ForCausalLM, DeepseekV4ForCausalLM [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM the same four rows plus GlmMoeDsaForCausalLM (GLM-5), Glm4MoeForCausalLM (GLM-4.5/4.6/4.7), KimiK3ForConditionalGeneration and KimiLinearForCausalLM [[mdl2-trtllm-supported-models]]; SGLang generative table names DeepSeek v1/v2/v3/R1 and Kimi K2 as supported [[mdl2-sglang-generative-models]]; llama.cpp reaches MLA through the DEEPSEEK2/DEEPSEEK32/DEEPSEEK4 arch entries, which are in its supports_sm_tensor == false list, i.e. no tensor-core matmul path for the attention itself [[mdl2-llamacpp-arch-class-registry]]; ik-llama-cpp records MLA as a feature that landed there first and was later upstreamed [[ik-llama-cpp]].ENGINE KERNEL CONSTRAINTS THAT CHANGE WHAT THE ENGINE MUST STORE (this is the important class of fact): (1) FlashInfer, which is SGLang's default attention backend and selectable in vLLM [[flashinfer]], is where the MLA decode kernels live, and they are instantiated per KV cache page-block size: gotcha-sm120-sparse-mla-block-size records vLLM computing compressed-layer pbs = block_size // compress_ratio (= 32 at block_size 64 for a ratio-2 layer) while FlashInfer's SM120 planner hard-codes _PAGE_BLOCK_SIZE = 64, so DeepSeek-V4.1-Flash will not start on 8x RTX PRO 6000 at TP8 at ANY global block size (64 gives pbs=32 and crashes; 128 gives 'No common block size'). Workaround is SM100 datacenter Blackwell or a FlashInfer planner change. (2) FlashMLA's 2026.09.30 release requires SM100/SM103 and CUDA >= 13.1, REMOVED Hopper support and removed V3/V3.2/V4.0, and supports ONLY FP8 and FP4 KV cache - unquantized bfloat16 KV is explicitly not supported [[kern-flashmla-github]]. So the kernel choice decides the KV CACHE FORMAT: to use that decode kernel the engine must store an FP8/FP4 latent with per-token inlined scales, and V4.1's layout is 528 bytes/token (512 of float8_e4m3 over all 512 dims - the 64 RoPE dims are quantized too, no bf16 part - plus 16 bytes of float8_e8m0 scales, one per 32 e4m3 values), with an fp4 variant at 288 bytes/token [[flashmla-decode-fp8-fp4-kv]]. A Hopper box therefore cannot take that kernel and an engine that only offers bf16 MLA KV cannot take it either. (3) Independent of the kernel: TRT-LLM's MLA rows carry two footnotes that constrain serving rather than correctness - chunked prefill for MLA only enables on SM90/SM100/SM103/SM107/SM120, and KV cache reuse for MLA only enables on SM90/100/103/107/120/121 in BF16/FP8 [[mdl2-trtllm-supported-models]]. (4) TRT-LLM states DeepSeek-V4 is Blackwell SM100+ only and Kimi K3 is SM100 family only, with a worked recipe split (DEP16 needs GB300-class per-GPU memory, TEP16 validated on GB200) [[mdl2-trtllm-supported-models]]. ISA CONSEQUENCE: at 68.6 KiB/token MLA moves the bottleneck off KV and onto weight streaming, so MLA-capable engines find their paged-attention machinery largely idle and their GQA-era cache-pressure tuning irrelevant [[mla-latent-attention]]. HARD BLOCKER, architecture-specific: TRT-LLM states DeepSeek-V4 is supported only on Blackwell GPUs (SM100+) [[mdl2-trtllm-supported-models]], and on SM120 consumer parts this checkpoint family hits a cluster of independent kernel gaps - FlashInfer's SM120 dispatch planner hard-codes _PAGE_BLOCK_SIZE = 64 while vLLM computes compressed-layer pbs = block_size // compress_ratio = 32, so there is no global block size that works, plus indexer_kv_dtype mxfp4 is rejected on SM120 ('requires Blackwell datacenter GPUs sm_10x') and had to be switched to fp8 [[sm120-sparse-mla-block-size]]. TRT-LLM's key-models matrix is the one place that records 'Sliding Window Attention: Yes' for a DeepSeek row, which is the compressed/hashed KV sliding_window 128 in this record [[mdl2-trtllm-supported-models]]. CONTEXT_SCALING_METHOD (yarn): config-derived; the existing free text 'YaRN (factor 16, original 65536)' agrees. SCHEME RECLASSIFIED (2026-10-04): was 'MLA variant with compressed/hashed KV, kv_lora_rank absent from config' as free text in attention_variant with no field for it. config genuinely has NO kv_lora_rank and no qk_nope_head_dim / v_head_dim; the only MLA dimension present is qk_rope_head_dim 64, plus q_lora_rank 1024, o_lora_rank 1024, head_dim 512 and index_head_dim 128 for the sparse indexer. Forcing the DeepSeek-V3 MLA numbers here would overstate KV cache by orders of magnitude, so kv_lora_rank is left null and the compression is recorded in kv_compression instead: a per-layer compress_ratios array of length 43 = num_hidden_layers. Values are 0 / 4 / 128 only: index 0 is 0 (first layer uncompressed), indices 1-2 are 0, then from index 3 onward the array ALTERNATES 4 and 128 (4,128,4,128,...) with the last index (42) back to 0. The 4-ratio layers are the CSA (compressed-sparse attention) layers and the 128-ratio layers the HCA baseline, plus num_hash_layers 3; sliding_window 128 and index_topk 512. A single averaged ratio is meaningless for this model - the two groups differ by 32x - which is exactly why the field is per-layer and named.