Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences

Understand

Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data (usually one chosen and rejected response pair per user prompt) to align LLMs to human preferences.

• In practice, multiple responses can exist for a given prompt with varying quality relative to each other.
• With availability of such quality ratings for multiple responses, we propose utilizing these responses to create multiple preference pairs for a given prompt.
• Our work focuses on systematically using the constructed multiple preference pair in DPO training via curriculum learning methodology.