arXiv:2310.02410 Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness

The headline MoE-quant claim with numbers: weight-only 2-bit applied ONLY to expert weights. Expert layers in MoE models are much more robust to quantization than conventional FFN layers; MoE with 2-bit expert weights can beat the dense model trained on the same dataset; model size cut 79.6% vs fp16; 1.24x speed-up on A100 with an optimized runtime.