Sparse Checkpointing for Fast and Reliable MoE Training

Understand

As large language models scale, training them requires thousands of GPUs over extended durations--making frequent failures an inevitable reality.

• While checkpointing remains the primary fault-tolerance mechanism, existing methods fall short when applied to Mixture-of-Experts (MoE) models.
• Due to their substantially larger training state, MoE models exacerbate checkpointing overheads, often causing costly stalls or prolonged recovery that severely degrade training efficiency.
• We present MoEvement, a distributed, in-memory checkpointing system tailored for MoE models.