Model Merging Scaling Laws in Large Language Models

Understand

We study empirical scaling laws for language model merging measured by cross-entropy.

• Despite its wide practical use, merging lacks a quantitative rule that predicts returns as we add experts or scale the model size.
• We identify a compact power law that links model size and expert number: the size-dependent floor decreases with model capacity, while the merging tail exhibits clear diminishing returns in the number of experts.
• The law holds in-domain and cross-domain, tightly fits measured curves across diverse architectures and methods (Average, TA, TIES, DARE), and explains two robust regularities: most gains arrive early, and variability shrinks as more experts are included.