Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models

Understand

Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective.

• In reality, humans express multiple, often conflicting objectives, such as helpfulness and harmlessness, with no natural scalarization.
• We study the multi-objective preference alignment problem, where a policy must balance several objectives simultaneously.
• We propose Multi-Objective Preference Optimization (MOPO), a constrained KL-regularized framework that maximizes a primary objective while enforcing lower bounds on secondary objectives via tunable safety thresholds.