Fetching the paper…

Reading the bibliography…

2025

Uncertainty-Aware Step-wise Verification with Generative Reward Models

Ye, Zihuiwen, Melo, Luckeciano Carvalho, Kaddar, Younesse et al.

Original
PDF
Post on X

Understand

Complex multi-step reasoning tasks, such as solving mathematical problems, remain challenging for large language models (LLMs).

While outcome supervision is commonly used, process supervision via process reward models (PRMs) provides intermediate rewards to verify step-wise correctness in solution traces.
However, as proxies for human judgement, PRMs suffer from reliability issues, including susceptibility to reward hacking.
In this work, we propose leveraging uncertainty quantification (UQ) to enhance the reliability of step-wise verification with generative reward models for mathematical reasoning tasks.

Uncertainty-Aware Step-wise Verification with Generative Reward Models · Around

Built on

On a Measure of the Information Provided by an Experiment
D. V. Lindley · 1956
Earlier work this paper cites.
Deep bayesian active learning with image data
Yarin Gal, Riashat Islam, and Zoubin Ghahramani · 2017
Earlier work this paper cites.
On calibration of modern neural networks
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger · 2017
Earlier work this paper cites.
Uncertainty estimation in autoregressive structured prediction
Original
Andrey Malinin and Mark Gales · 2020
Earlier work this paper cites.
Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021
Earlier work this paper cites.
Measuring mathematical problem solving with the math dataset
Original
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt · 2021
Earlier work this paper cites.
Language models (mostly) know what they know
Original
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al · 2022
Earlier work this paper cites.
Solving math word problems with process-and outcome-based feedback
Original
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins · 2022
Earlier work this paper cites.
Self-consistency improves chain of thought reasoning in language models
Original
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou · 2022
Earlier work this paper cites.

Similar

Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Cited alongside, same era.
How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks
Original
Luís Felipe P Cattelan and Danilo Silva · 2023
Cited alongside, same era.
Let’s verify step by step
Original
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe · 2023
Cited alongside, same era.
Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization
Original
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al · 2023
Cited alongside, same era.
Judging llm-as-a-judge with mt-bench and chatbot arena

Then

Nv-embed: Improved techniques for training llms as generalist embedding models
Original
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping · 2024
Later among the works it cites.
Iterative reasoning preference optimization
Original
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston · 2024
Later among the works it cites.
A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions, 2024
Original
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z. Ren, and Anirudha Majumdar · 2024
Later among the works it cites.
Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024
Original
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al · 2023

Cited alongside, same era.

Deep bayesian active learning for preference modeling in large language models

Luckeciano Carvalho Melo, Panagiotis Tigas, Alessandro Abate, and Yarin Gal · 2024

Cited alongside, same era.

Detecting hallucinations in large language models using semantic entropy

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal · 2024

Cited alongside, same era.

Improving uncertainty quantification in large language models via semantic embeddings

Yashvir S Grewal, Edwin V Bonilla, and Thang D Bui · 2024

Cited alongside, same era.

Semantic entropy probes: Robust and cheap hallucination detection in llms

Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal · 2024

Cited alongside, same era.

Improving reward models with synthetic critiques, 2024

Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, and Matthias Gallé · 2024

Later among the works it cites.

Generative verifiers: Reward modeling as next-token prediction, 2024

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal · 2024

Later among the works it cites.

Processbench: Identifying process errors in mathematical reasoning

Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin · 2024

Later among the works it cites.

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

DeepSeek-AI · 2025

Closest in time.

The lessons of developing process reward models in mathematical reasoning

Original

Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin · 2025

Closest in time.