LaDiMo: Layer-wise Distillation Inspired MoEfier

Understand

The advent of large language models has revolutionized natural language processing, but their increasing complexity has led to substantial training costs, resource demands, and environmental impacts.

• In response, sparse Mixture-of-Experts (MoE) models have emerged as a promising alternative to dense models.
• Since training MoE models from scratch can be prohibitively expensive, recent studies have explored leveraging knowledge from pre-trained non-MoE models.
• However, existing approaches have limitations, such as requiring significant hardware resources and data.