ROPO: Robust Preference Optimization for Large Language Models

Understand

Preference alignment is pivotal for empowering large language models (LLMs) to generate helpful and harmless responses.

• However, the performance of preference alignment is highly sensitive to the prevalent noise in the preference data.
• Recent efforts for this problem either marginally alleviate the impact of noise without the ability to actually reduce its presence, or rely on costly teacher LLMs prone to reward misgeneralization.
• To address these challenges, we propose the RObust Preference Optimization (ROPO) framework, an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models.