m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers

Understand

Modular neural architectures are gaining attention for their powerful generalization and efficient adaptation to new domains.

• However, training these models poses challenges due to optimization difficulties arising from intrinsic sparse connectivity.
• Leveraging knowledge from monolithic models through techniques like knowledge distillation can facilitate training and enable integration of diverse knowledge.
• Nevertheless, conventional knowledge distillation approaches are not tailored to modular models and struggle with unique architectures and enormous parameter counts.