InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Understand

Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods.

• Existing works on model fusion focus primarily on supervised fine-tuning (SFT), leaving preference alignment (PA) --a critical phase for enhancing LLM performance--largely unexplored.
• The current few fusion methods on PA phase, like WRPO, simplify the process by utilizing only response outputs from source models while discarding their probability information.
• To address this limitation, we propose InfiFPO, a preference optimization method for implicit model fusion.