ModuleFormer: Modularity Emerges from Mixture-of-Experts

Understand

Large Language Models (LLMs) have achieved remarkable results.

• However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge.
• This paper proposes a new neural network architecture, ModuleFormer, that leverages modularity to improve the efficiency and flexibility of large language models.
• ModuleFormer is based on the Sparse Mixture of Experts (SMoE).