LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

Understand

Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5.

• However, systematic research on MoE remains severely constrained by the prohibitive computational costs of training and evaluation, restricting large-scale studies accessible to most researchers.
• We introduce LibMoE, a unified framework for reproducible, efficient, and extensible MoE research that supports both pretraining and sparse-upcycling regimes.
• Beyond unified implementations, the framework provides transparent analytical tools for probing routing and expert dynamics.