Let's Verify Step by Step

Understand

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning.

• However, even state-of-the-art models still regularly produce logical mistakes.
• To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step.
• Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods.