Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

Understand

Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach.

• However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses.
• This results in conflicting optimization directions, hindering the optimization on the Pareto Front.
• To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts.