MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router

Understand

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts.

• Pruning MoE can reduce network weights while maintaining model performance.
• Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron.
• Our pruning method is one-shot, requiring no retraining or weight updates.