Weak-to-Strong Generalization via Bregman Bias-Variance Decomposition

Understand

Weak-to-strong generalization (W2SG) is the phenomenon in which a powerful student model, trained on labels produced by a weaker teacher, ultimately outperforms the teacher on the target task.

• In this work, we theoretically investigate how W2SG can arise via a generalized bias-variance decomposition under Bregman divergence.
• We show that the expected population risk gap between the student and the teacher is characterized by the expected misfit between the two models.
• Unlike earlier misfit-based analyses, our theory removes several restrictive assumptions, e.g., it does not require the student hypothesis class to be convex.