Meta-Learning Objectives for Preference Optimization

Understand

Evaluating preference optimization (PO) algorithms on LLM alignment is a challenging task that presents prohibitive costs, noise, and several variables like model size and hyper-parameters.

• In this work, we show that it is possible to gain insights on the efficacy of PO algorithm on simpler benchmarks.
• We design a diagnostic suite of MuJoCo tasks and datasets, which we use to systematically evaluate PO algorithms, establishing a more controlled and cheaper benchmark.
• We then propose a novel family of PO algorithms based on mirror descent, which we call Mirror Preference Optimization (MPO).