$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

Understand

Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences.

• This paper introduces $f$-divergence Preference Optimization ($f$-PO), a novel framework that generalizes and extends existing approaches.
• $f$-PO minimizes $f$-divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences.
• Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of $f$-divergences.