Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Understand

Direct Preference Optimisation (DPO) is effective at significantly improving the performance of large language models (LLMs) on downstream tasks such as reasoning, summarisation, and alignment.

• Using pairs of preferred and dispreferred data, DPO models the relative probability of picking one response over another.
• In this work, first we show theoretically that the standard DPO loss can lead to a reduction of the model's likelihood of the preferred examples, as long as the relative probability between the preferred and dispreferred classes increases.
• We then show empirically that this phenomenon occurs when fine-tuning LLMs on common datasets, especially datasets in which the edit distance between pairs of completions is low.