Less is More: Improving LLM Alignment via Preference Data Selection

Understand

Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences.

• While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection.
• Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing a novel margin-maximization principle for dataset curation in DPO training.
• To further mitigate the noise in different reward models, we propose a Bayesian Aggregation approach that unifies multiple margin sources (external and implicit) into a single preference probability.