OLMoE: Open Mixture-of-Experts Language Models

MoE research was closed: without released weights, data and training logs, nobody could audit whether a sparse model's routing actually specialises.

Not a new MoE mechanism. It is the fully open instantiation of the Switch-style recipe at 7B total and 1B active per token (about 14.3 percent active, 7x sparse), pre-trained on 5T tokens. Routing is token-choice top-1 with an auxiliary load-balancing loss, the Switch/GShard family. The contribution is evidence: weights, code, training data (OLMo-mix-1124) and full training logs are all released, and the routing analysis shows high expert specialisation with low inter-expert similarity. That auditability is the point; it let the community see what token-choice top-1 actually converges to.

No speedup claim. The claim is quality per active parameter: OLMoE-1B-7B-0924 outperforms all available models with similar ACTIVE parameter counts and surpasses larger dense models including Llama-2-13B-Chat and DeepSeekMoE-16B. The 1B-active over 7B-total ratio is the architecture-relevant number for serving planning.

Fully open (code, data, weights, logs), which is unusual and is the reason this paper is in the knowledge base at all: it is the reference implementation against which closed MoE claims can be checked. Verified as OlmoeForCausalLM in vLLM's native supported-models list. Load balancing is auxiliary-loss, NOT auxiliary-loss-free. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'iclr', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'ICLR' paper page: ICLR 2025 poster 27777. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2409.02060). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). YEAR FIELD CORRECTED 2026-10-04 (same pass as the venue correction above): year 2024 -> 2025 to match the ICLR 2025 venue year. The prior value 2024 is the arXiv posting year and remains true of the preprint. status stays contested. This drift is why the venue correction and the year correction were done together rather than separately: setting venue without setting year leaves the record internally inconsistent. 2026-10-10: ICLR 2025 accepted titles index (idx7.py: https://iclr.cc/virtual/2025/papers.html?filter=titles) lists "OLMoE: Open Mixture-of-Experts Language Models". arXiv API 2409.02060 (axc.py) comment contains page counts only. Source pap-arch-2409-02060 cites arXiv preprint vs ICLR 2025 index. Sources disagree; status remains contested.