Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Understand

The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost.

• However, conventional MoEs pose challenges at scale due to the need to store all experts in memory.
• In this paper, we push MoE to the limit.
• We propose extremely parameter-efficient MoE by uniquely combining MoE architecture with lightweight experts.Our MoE architecture outperforms standard parameter-efficient fine-tuning (PEFT) methods and is on par with full fine-tuning by only updating the lightweight experts -- less than 1% of an 11B parameters model.