Subspace-Boosted Model Merging

Understand

Model merging enables the combination of multiple specialized expert models into a single model capable of performing multiple tasks.

• However, the benefits of merging an increasing amount of specialized experts generally lead to diminishing returns and reduced overall performance gains.
• In this work, we empirically and theoretically analyze this limitation, proving that for Task Arithmetic-based methods, as more experts are merged, the common information dominates the task-specific information, leading to inevitable rank collapse.
• To mitigate this issue, we introduce Subspace Boosting, which operates on the singular value decomposed task vector space and maintains task vector ranks.