Model Merging in Pre-training of Large Language Models

Understand

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored.

• In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process.
• Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior.
• These improvements lead to both more efficient model development and significantly lower training costs.