Mixture of A Million Experts (PEER layers)

VENDOR BENCHMARK CLAIM. PEER authors claim 'PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off.' SKEPTICAL CONSIDERATIONS: (1) The paper is a single-author work (Xu Owen He) which may indicate limited peer review. (2) The 'over a million experts' claim is for a specific layer design (product key retrieval) that may not generalize to all MoE architectures. (3) The comparison is against 'coarse-grained MoEs' - the specific baseline models and configurations are not detailed in the abstract. (4) The efficiency claims are for language modeling tasks - generalization to other tasks is not demonstrated. (5) Self-benchmark by the PEER authors. 2026-10-10: Fetched https://arxiv.org/abs/2407.04153. Title: 'Mixture of A Million Experts'. Preprint matching record claims; abstract confirms PEER architecture scaling experts to over a million. Status remains contested due to single-author / self-benchmark considerations.