Knowledge Divergence and the Value of Debate for Scalable Oversight

Understand

AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage.

• We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models.
• Using principal angles between models' representation subspaces, we prove that the debate advantage admits an exact closed form.
• When models share identical training corpora, debate reduces to RLAIF-like where a single-agent method recovers the same optimum.