Data-Centric Human Preference with Rationales for Direct Preference Alignment

Understand

Aligning language models with human preferences through reinforcement learning from human feedback is crucial for their safe and effective deployment.

• The human preference is typically represented through comparison where one response is chosen over another for a given prompt.
• However, standard preference datasets often lack explicit information on why a particular choice was made, presenting an ambiguity that can hinder efficient learning and robust alignment, especially given the high cost of acquiring extensive human annotations.
• While many studies focus on algorithmic improvements, this work adopts a data-centric perspective, exploring how to enhance learning from existing preference data.