Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Understand

Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences.

• Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance.
• To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking.
• To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO.