Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast (NeurIPS 2024)

COUNTER-EVIDENCE TO MoE EFFICIENCY CLAIMS. This paper challenges the standard MoE assumption that only activated experts matter. Key findings: (1) 'Increasing the number of activated experts does not necessarily improve and can even degrade the output quality' - this contradicts the common belief that more experts = better quality. (2) 'Output distributions from an MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically.' (3) The proposed SCMoE method improves Mixtral 8x7B accuracy on GSM8K from 61.79 to 66.94. SKEPTICAL CONSIDERATIONS: (1) The degradation finding is for specific models and tasks - it may not generalize to all MoE models. (2) The SCMoE method adds computational overhead (contrasting strong and weak activation) which may negate the efficiency gains. (3) Self-benchmark by the SCMoE authors. This paper is important because it shows that MoE routing is not well-understood and that simple 'more experts' heuristics can fail.