Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing

Understand

Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one.

• However, parameter conflicts between models leads to performance degradation in averaging.
• While model routing addresses this issue by selecting individual models during inference, it imposes excessive storage and compute costs, and fails to leverage the common knowledge from different models.
• In this work, we observe that different layers exhibit varying levels of parameter conflicts.