Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach

Understand

Aligning large language models (LLMs) with human values and safety constraints is challenging, especially when objectives like helpfulness, truthfulness, and avoidance of harm conflict.

• Reinforcement Learning from Human Feedback (RLHF) has achieved notable success in steering models, but is complex and can be unstable.
• Recent approaches such as Direct Preference Optimization (DPO) simplify preference-based fine-tuning but may introduce bias or trade-off certain objectives~\cite{dpo}.
• In this work, we propose a Group Relative Policy Optimization (GRPO) framework with a multi-label reward regression model to achieve safe and aligned language generation.