Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models

Understand

Weak-to-Strong generalization (W2SG) is a new trend to elicit the full capabilities of a strong model with supervision from a weak model.

• While existing W2SG studies focus on simple tasks like binary classification, we extend this paradigm to complex interactive decision-making environments.
• Specifically, we fine-tune a strong model with trajectories of intermediate actions generated by a weak model.
• Motivated by the human learning process, we propose to generalize not only success knowledge but also failure experience so that the strong model can learn from failed trajectories accumulated by weak models.