Built on

Ape210k: A large-scale and template-rich dataset of math word problems
Original
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu · 2020
Earlier work this paper cites.
Neural-symbolic solver for math word problems with auxiliary tasks
Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin · 2021
Earlier work this paper cites.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks
Original
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen · 2022
Earlier work this paper cites.
Self-critiquing models for assisting human evaluators
Original
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Generating sequences by learning to self-correct
Original
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi · 2022
Earlier work this paper cites.
Star: Bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman · 2022
Earlier work this paper cites.
Let’s verify step by step
Original
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe · 2023
Earlier work this paper cites.
Cmath: can your language model pass chinese elementary school math test?
Original
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang · 2023
Earlier work this paper cites.
Introducing meta llama 3: The most capable openly available llm to date
AI@Meta · 2024
Earlier work this paper cites.

Similar

Large language monkeys: Scaling inference compute with repeated sampling
Original
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini · 2024
Cited alongside, same era.
Stepwise verification and remediation of student reasoning errors with large language model tutors
Original
Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan · 2024
Cited alongside, same era.
ToRA: A tool-integrated reasoning agent for mathematical problem solving
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen · 2024
Cited alongside, same era.
Training language models to self-correct via reinforcement learning
Original
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al · 2024
Cited alongside, same era.

Then

Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn · 2024
Closest in time.
Bond: Aligning llms with best-of-n distillation
Original
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al · 2024
Closest in time.
Scaling llm test-time compute optimally can be more effective than scaling model parameters
Original
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024
Closest in time.
Toward self-improvement of llms via imagination, searching, and criticizing
Original
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu · 2024
Closest in time.
Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Teaching LLMs to Refine with Tools

Yu, Dian, Zhang, Yuheng, Xu, Jiahao et al.

Understand

Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement.

However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors.
We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs.
CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants.

Built on

Ape210k: A large-scale and template-rich dataset of math word problems
Original
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu · 2020
Earlier work this paper cites.
Neural-symbolic solver for math word problems with auxiliary tasks
Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin · 2021
Earlier work this paper cites.
Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks
Original
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen · 2022
Earlier work this paper cites.
Self-critiquing models for assisting human evaluators
Original
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Generating sequences by learning to self-correct
Original
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi · 2022
Earlier work this paper cites.
Star: Bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman · 2022
Earlier work this paper cites.
Let’s verify step by step
Original
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe · 2023
Earlier work this paper cites.
Cmath: can your language model pass chinese elementary school math test?
Original
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang · 2023
Earlier work this paper cites.
Introducing meta llama 3: The most capable openly available llm to date
AI@Meta · 2024
Earlier work this paper cites.

Similar

Large language monkeys: Scaling inference compute with repeated sampling
Original
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini · 2024
Cited alongside, same era.
Stepwise verification and remediation of student reasoning errors with large language model tutors
Original
Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan · 2024
Cited alongside, same era.
ToRA: A tool-integrated reasoning agent for mathematical problem solving
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen · 2024
Cited alongside, same era.
Training language models to self-correct via reinforcement learning
Original
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al · 2024
Cited alongside, same era.

Then

Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn · 2024
Closest in time.
Bond: Aligning llms with best-of-n distillation
Original
Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al · 2024
Closest in time.
Scaling llm test-time compute optimally can be more effective than scaling model parameters
Original
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024
Closest in time.
Toward self-improvement of llms via imagination, searching, and criticizing
Original
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu · 2024
Closest in time.
Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer

Original

Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang · 2024

Cited alongside, same era.

Self-refine: Iterative refinement with self-feedback

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al · 2024

Cited alongside, same era.

Adaptive inference-time compute: Llms can predict if they can do better, even mid-generation

Original

Rohin Manvi, Anikait Singh, and Stefano Ermon · 2024

Cited alongside, same era.

Llm critics help catch llm bugs

Original

Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike · 2024

Cited alongside, same era.

Iterative reasoning preference optimization

Original

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston · 2024

Cited alongside, same era.

Recursive introspection: Teaching language model agents how to self-improve

Original

Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar · 2024

Cited alongside, same era.

Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning

Original

Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li

Cited in the paper.

Shepherd: A critic for language model generation

Original

Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz

Cited in the paper.

Original

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, et al · 2024

Closest in time.

Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems

Original

Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu · 2024

Closest in time.

Siam: Self-improving code-assisted mathematical reasoning of large language models

Original

Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, and Dong Yu · 2024

Closest in time.