DeepSeek-V3 Technical Report · inference-research

Fields

id

sources/attn-deepseek-v3-paper

URL

https://arxiv.org/abs/2412.19437

Publisher

arXiv

Kind

paper

Published

2024-12-27

Accessed

2026-10-04

notes

DeepSeek-V3 paper (arXiv 2412.19437). Uses MLA with kv_lora_rank=512, qk_rope_head_dim=64, 61 layers. MLA cache = 61*(512+64) = 35,136 elements/token = 70,272 bytes = 68.6 KiB/token at bf16. vs MHA equivalent: 61*2*128*(128+64) = 2,998,272 elements = 5,856 KiB/token. 85.3x reduction.

Notes

DeepSeek-V3 paper (arXiv 2412.19437). Uses MLA with kv_lora_rank=512, qk_rope_head_dim=64, 61 layers. MLA cache = 61*(512+64) = 35,136 elements/token = 70,272 bytes = 68.6 KiB/token at bf16. vs MHA equivalent: 61*2*128*(128+64) = 2,998,272 elements = 5,856 KiB/token. 85.3x reduction.

Cross-references

This record references no other records.