Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

Understand

Automated red teaming can discover rare model failures and generate challenging examples that can be used for training or evaluation.

• However, a core challenge in automated red teaming is ensuring that the attacks are both diverse and effective.
• Prior methods typically succeed in optimizing either for diversity or for effectiveness, but rarely both.
• In this paper, we provide methods that enable automated red teaming to generate a large number of diverse and successful attacks.