Learning to Better Search with Language Models via Guided Reinforced Self-Training

Understand

While language models have shown remarkable performance across diverse tasks, they still encounter challenges in complex reasoning scenarios.

• Recent research suggests that language models trained on linearized search traces toward solutions, rather than solely on the final solutions, exhibit improved generalization, despite the search traces being potentially noisy or suboptimal.
• However, relying on such imperfect traces can result in inefficient use of test-time compute.
• To address this, we propose guided reinforced self-training (Guided-ReST), a fine-tuning algorithm designed to improve the model's capability for effective search during inference.