Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

Understand

Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities.

• This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats.
• To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment.
• A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes.