DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Fields

id

sources/arch-deepseekmoe

URL

https://arxiv.org/abs/2401.06066

Publisher

DeepSeek-AI (arXiv)

Kind

paper

Published

2024-01-11

Accessed

2026-10-03

notes

Defines fine-grained expert segmentation (split the FFN intermediate hidden dimension, activate more, smaller experts at constant compute) and shared-expert isolation (always-on experts capture common knowledge, reducing redundancy among routed experts).

Notes

Defines fine-grained expert segmentation (split the FFN intermediate hidden dimension, activate more, smaller experts at constant compute) and shared-expert isolation (always-on experts capture common knowledge, reducing redundancy among routed experts).

Cross-references

This record references no other records.