Activation-Guided Consensus Merging for Large Language Models

Understand

Recent research has increasingly focused on reconciling the reasoning capabilities of System 2 with the efficiency of System 1.

• While existing training-based and prompt-based approaches face significant challenges in terms of efficiency and stability, model merging emerges as a promising strategy to integrate the diverse capabilities of different Large Language Models (LLMs) into a unified model.
• However, conventional model merging methods often assume uniform importance across layers, overlooking the functional heterogeneity inherent in neural components.
• To address this limitation, we propose \textbf{A}ctivation-Guided \textbf{C}onsensus \textbf{M}erging (\textbf{ACM}), a plug-and-play merging framework that determines layer-specific merging coefficients based on mutual information between activations of pre-trained and fine-tuned models.