MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

Reformulates MoE as block-sparse operations with block-sparse kernels, so token dynamism is handled without dropping tokens or padding to the max expert size. Up to 40% end-to-end speedup over Tutel and 2.4x over Megatron-LM. Primarily a training result; cited here for the block-sparse formulation whose kernels also apply to MoE inference.