A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

Understand

With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical.

• Direct Preference Optimization (DPO) has emerged as a promising approach for alignment, acting as an RL-free alternative to Reinforcement Learning from Human Feedback (RLHF).
• Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature.
• In this work, we present a comprehensive review of the challenges and opportunities in DPO, covering theoretical analyses, variants, relevant preference datasets, and applications.