Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models

Understand

Self-Rewarding Language Models (SRLMs) achieve notable success in iteratively improving alignment without external feedback.

• Yet, despite their striking empirical progress, the core mechanisms driving their capabilities remain unelucidated, leaving a critical gap in theoretical understanding.
• This paper provides the first rigorous theoretical guarantees for SRLMs.
• We first establish a lower bound that characterizes the fundamental limits of a single update step, revealing a critical dependence on the quality of the initial model.