Why Do More Experts Fail? A Theoretical Analysis of Model Merging

Understand

Model merging dramatically reduces storage and computational resources by combining multiple expert models into a single multi-task model.

• Although recent model merging methods have shown promising results, they struggle to maintain performance gains as the number of merged models increases.
• In this paper, we investigate the key obstacles that limit the scalability of model merging when integrating a large number of expert models.
• First, we prove that there is an upper bound on model merging.