DeepSeek-V3 Technical Report · inference-research

Fields

id

sources/arch-deepseek-v3

URL

https://arxiv.org/abs/2412.19437

Publisher

DeepSeek-AI (arXiv)

Kind

paper

Published

2024-12-27

Accessed

2026-10-03

notes

Section 4.2 hyper-parameters: 61 layers, hidden 7168, n_h=128, d_h=128, d_c=512, d'_c=1536, d_rh=64, 1 shared + 256 routed experts, 8 activated, at most 4 nodes per token, first 3 layers dense FFN. Section 2.1.2 auxiliary-loss-free load balancing via a per-expert bias term updated by +/-gamma. 671B total / 37B active.

Notes

Section 4.2 hyper-parameters: 61 layers, hidden 7168, n_h=128, d_h=128, d_c=512, d'_c=1536, d_rh=64, 1 shared + 256 routed experts, 8 activated, at most 4 nodes per token, first 3 layers dense FFN. Section 2.1.2 auxiliary-loss-free load balancing via a per-expert bias term updated by +/-gamma. 671B total / 37B active.

Cross-references

This record references no other records.