Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

Understand

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing.

• They effectively expand the model capacity while incurring a minimal increase in computation cost during training.
• However, deploying such models for inference is difficult due to their large size and complex communication pattern.
• In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment.