Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts

Understand

Efficiency, specialization, and adaptability to new data distributions are qualities that are hard to combine in current Large Language Models.

• The Mixture of Experts (MoE) architecture has been the focus of significant research because its inherent conditional computation enables such desirable properties.
• In this work, we focus on "upcycling" dense expert models into an MoE, aiming to improve specialization while also adding the ability to adapt to new tasks easily.
• We introduce Nexus, an enhanced MoE architecture with adaptive routing where the model learns to project expert embeddings from domain representations.