From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging

Understand

Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.

• This has led to a proliferation of expert models and adapters, often shared via platforms like HuggingFace and AdapterHub.
• Model merging has recently emerged as an effective way to leverage these existing resources, enabling the composition of capabilities from different model checkpoints.
• A natural pipeline has thus formed to harness the benefits of transfer learning and amortize sunk training costs: models are pre-trained on general data, fine-tuned on specific tasks, and then multiple checkpoints are merged to obtain a more capable model.