DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Section 2.1 is the primary statement of MLA. Table 1 gives KV-cache-per-token in ELEMENTS: MHA 2*n_h*d_h*l, GQA 2*n_g*d_h*l, MQA 2*d_h*l, MLA (d_c+d_rh)*l ~= (9/2)*d_h*l. States d_c=4*d_h and d_rh=d_h/2, hence 'equal to GQA with only 2.25 groups'. Section 2.1.2: W_UK can be absorbed into W_Q and W_UV into W_O, so keys/values never need to be materialised for attention. Section 2.1.3: RoPE is incompatible with low-rank KV compression, hence decoupled RoPE. Section 2.2.2 device-limited routing; 2.2.3 three auxiliary losses; 2.2.4 token dropping with device capacity factor 1.0 and ~10% of sequences never dropped. Abstract: KV cache reduced 93.3% vs DeepSeek 67B, max generation throughput 5.76x.