DeepSeek-V2

The model that introduced MLA as the reason it is in this repo. kv_cache_bytes_per_token = 60 layers x (512 + 64) x 2 = 69,120 B/token (67.5 KiB), same latent formula as V3 and still 58x smaller than the naive GQA figure (60 x 128 x 128 x 2 x 2 = 3,932,160 B). Smaller than V3 in every axis: hidden 5120, 60 layers, 160 routed + 2 shared experts, 6 per token, intermediate 12288, moe_intermediate 1536, first_k_dense_replace 1. Vendor figures 236B total / 21B activated; our arithmetic gives 235.7B / 21.3B. Interesting serving artifact: rope_scaling mscale is 0.707 here vs 1.0 in V3, so V2 needed an attention temperature correction that V3 dropped. YaRN factor 40 over 4096. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=mla. REQUIRED KERNEL CLASS: an MLA decode kernel. MLA is not a variant of the GQA kernel - the cache is a shared low-rank latent (d_c) plus a decoupled RoPE key, not per-head K/V, so an engine that lacks a dedicated MLA decode path cannot serve the class at all. SUPPORTED BY (cited): vLLM native rows DeepseekV2ForCausalLM, DeepseekV3ForCausalLM, DeepseekV32ForCausalLM, DeepseekV4ForCausalLM [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM the same four rows plus GlmMoeDsaForCausalLM (GLM-5), Glm4MoeForCausalLM (GLM-4.5/4.6/4.7), KimiK3ForConditionalGeneration and KimiLinearForCausalLM [[mdl2-trtllm-supported-models]]; SGLang generative table names DeepSeek v1/v2/v3/R1 and Kimi K2 as supported [[mdl2-sglang-generative-models]]; llama.cpp reaches MLA through the DEEPSEEK2/DEEPSEEK32/DEEPSEEK4 arch entries, which are in its supports_sm_tensor == false list, i.e. no tensor-core matmul path for the attention itself [[mdl2-llamacpp-arch-class-registry]]; ik-llama-cpp records MLA as a feature that landed there first and was later upstreamed [[ik-llama-cpp]].ENGINE KERNEL CONSTRAINTS THAT CHANGE WHAT THE ENGINE MUST STORE (this is the important class of fact): (1) FlashInfer, which is SGLang's default attention backend and selectable in vLLM [[flashinfer]], is where the MLA decode kernels live, and they are instantiated per KV cache page-block size: gotcha-sm120-sparse-mla-block-size records vLLM computing compressed-layer pbs = block_size // compress_ratio (= 32 at block_size 64 for a ratio-2 layer) while FlashInfer's SM120 planner hard-codes _PAGE_BLOCK_SIZE = 64, so DeepSeek-V4.1-Flash will not start on 8x RTX PRO 6000 at TP8 at ANY global block size (64 gives pbs=32 and crashes; 128 gives 'No common block size'). Workaround is SM100 datacenter Blackwell or a FlashInfer planner change. (2) FlashMLA's 2026.09.30 release requires SM100/SM103 and CUDA >= 13.1, REMOVED Hopper support and removed V3/V3.2/V4.0, and supports ONLY FP8 and FP4 KV cache - unquantized bfloat16 KV is explicitly not supported [[kern-flashmla-github]]. So the kernel choice decides the KV CACHE FORMAT: to use that decode kernel the engine must store an FP8/FP4 latent with per-token inlined scales, and V4.1's layout is 528 bytes/token (512 of float8_e4m3 over all 512 dims - the 64 RoPE dims are quantized too, no bf16 part - plus 16 bytes of float8_e8m0 scales, one per 32 e4m3 values), with an fp4 variant at 288 bytes/token [[flashmla-decode-fp8-fp4-kv]]. A Hopper box therefore cannot take that kernel and an engine that only offers bf16 MLA KV cannot take it either. (3) Independent of the kernel: TRT-LLM's MLA rows carry two footnotes that constrain serving rather than correctness - chunked prefill for MLA only enables on SM90/SM100/SM103/SM107/SM120, and KV cache reuse for MLA only enables on SM90/100/103/107/120/121 in BF16/FP8 [[mdl2-trtllm-supported-models]]. (4) TRT-LLM states DeepSeek-V4 is Blackwell SM100+ only and Kimi K3 is SM100 family only, with a worked recipe split (DEP16 needs GB300-class per-GPU memory, TEP16 validated on GB200) [[mdl2-trtllm-supported-models]]. ISA CONSEQUENCE: at 68.6 KiB/token MLA moves the bottleneck off KV and onto weight streaming, so MLA-capable engines find their paged-attention machinery largely idle and their GQA-era cache-pressure tuning irrelevant [[mla-latent-attention]]. CONTEXT_SCALING_METHOD (yarn): config-derived; the existing free text 'YaRN (factor 40, original 4096, mscale 0.707)' agrees. SOURCE CORRECTION (2026-10-05): this record cited model-cn-deepseek-v3-config (the DeepSeek-V3 config) as its config evidence, which is the wrong checkpoint for a V2 record - every field in that config disagrees with V2 on model_type, layer count, hidden size and expert count. The correct source record already existed in data/sources/ (arch-hf-config-deepseek-v2, https://huggingface.co/deepseek-ai/DeepSeek-V2/raw/main/config.json) but was not wired into this record, so it was an orphan. Re-read directly: config declares model_type deepseek_v2, num_hidden_layers 60, hidden_size 5120, kv_lora_rank 512, qk_rope_head_dim 64, qk_nope_head_dim 128, v_head_dim 128, num_attention_heads 128, num_key_value_heads 128, n_routed_experts 160, n_shared_experts 2, num_experts_per_tok 6, first_k_dense_replace 1, moe_intermediate_size 1536, intermediate_size 12288, rope_scaling type yarn factor 40 mscale 0.707. EVERY NUMBER IN THIS RECORD IS CONFIRMED against that config; none needed changing. kv_cache_bytes_per_token = 60 x (512 + 64) x 2 = 69,120 B/token (67.5 KiB). Note the trap that caused the mis-citation: V2 and V3 share kv_lora_rank 512, qk_rope_head_dim 64 and v_head_dim 128, so the KV GEOMETRY is identical between the two releases and reading it off V3's config gives the right answer for the wrong reason. Only num_layers (60 vs 61) and hidden_size (5120 vs 7168) would have exposed it - and the KV figure does depend on the layer count, so a V3-based read would have given 70,272, wrong by 1.7%. The record was right by independent derivation, not by this source, which is why the error was invisible until the sources were checked.