Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Understand

This study addresses the challenge of noise in training datasets for Direct Preference Optimization (DPO), a method for aligning Large Language Models (LLMs) with human preferences.

• We categorize noise into pointwise noise, which includes low-quality data points, and pairwise noise, which encompasses erroneous data pair associations that affect preference rankings.
• Utilizing Distributionally Robust Optimization (DRO), we enhance DPO's resilience to these types of noise.
• Our theoretical insights reveal that DPO inherently embeds DRO principles, conferring robustness to pointwise noise, with the regularization coefficient $\beta$ playing a critical role in its noise resistance.