DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Fields

id

sources/attn-deepseek-v2-mla-paper

URL

https://arxiv.org/abs/2405.04434

Publisher

arXiv

Kind

paper

Published

2024-05-07

Accessed

2026-10-04

notes

DeepSeek-V2 paper (arXiv 2405.04434). Introduces Multi-head Latent Attention (MLA). Table 1: KV cache per token in elements: MHA 2*n_h*d_h*l, GQA 2*n_g*d_h*l, MQA 2*d_h*l, MLA (d_c+d_rh)*l. With d_c=4*d_h and d_rh=d_h/2, MLA = 4.5*d_h*l, equal to GQA with only 2.25 groups. 93.3% KV cache reduction vs dense DeepSeek 67B, 5.76x max generation throughput. Absorption: W_UK into W_Q, W_UV into W_O.

Notes

DeepSeek-V2 paper (arXiv 2405.04434). Introduces Multi-head Latent Attention (MLA). Table 1: KV cache per token in elements: MHA 2*n_h*d_h*l, GQA 2*n_g*d_h*l, MQA 2*d_h*l, MLA (d_c+d_rh)*l. With d_c=4*d_h and d_rh=d_h/2, MLA = 4.5*d_h*l, equal to GQA with only 2.25 groups. 93.3% KV cache reduction vs dense DeepSeek 67B, 5.76x max generation throughput. Absorption: W_UK into W_Q, W_UV into W_O.

Cross-references

This record references no other records.