Offline Reinforcement Learning for LLM Multi-Step Reasoning

Fine-tuning language models from human preferences

Original

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019 · 1909

Earlier work this paper cites.

Rank analysis of incomplete block designs: I. the method of paired comparisons

Ralph Allan Bradley and Milton E Terry. 1952 · 1952

Earlier work this paper cites.

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Ronald J Williams. 1992 · 1992

Earlier work this paper cites.

Alfworld: Aligning text and embodied environments for interactive learning

Original

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020 · 2010

Earlier work this paper cites.

Modeling purposeful adaptive behavior with the principle of maximum causal entropy

Brian D Ziebart. 2010 · 2010

Earlier work this paper cites.

Reinforcement learning with deep energy-based policies

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. 2017 · 2017

Earlier work this paper cites.

Bridging the gap between value and policy based reinforcement learning

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. 2017 · 2017

Earlier work this paper cites.

Proximal policy optimization algorithms

Original

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 · 2017

Earlier work this paper cites.

Mastering chess and shogi by self-play with a general reinforcement learning algorithm

Original

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2017 · 2017

Earlier work this paper cites.

Breaking the softmax bottleneck: A high-rank rnn language model

Original

Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2017 · 2017

Earlier work this paper cites.

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Original

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018 · 2018

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021

Earlier work this paper cites.

Efficient (soft) q-learning for text generation with limited good data

Original

Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. 2021 · 2021

Earlier work this paper cites.

Measuring mathematical problem solving with the math dataset

Original

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Show your work: Scratchpads for intermediate computation with language models

Original

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021 · 2021

Earlier work this paper cites.

Language models as zero-shot planners: Extracting actionable knowledge for embodied agents

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

When should we prefer offline reinforcement learning over behavioral cloning?

Original

Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Solving math word problems with process-and outcome-based feedback

Original

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

React: Synergizing reasoning and acting in language models

Original

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022 · 2022

Earlier work this paper cites.

Star: Bootstrapping reasoning with reasoning

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Cited alongside, same era.

Rest meets react: Self-improvement for multi-step reasoning llm agent

Original

Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. 2023 · 2023

Cited alongside, same era.

Raft: Reward ranked finetuning for generative foundation model alignment

Original

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023 · 2023

Cited alongside, same era.

Alphazero-like tree-search can guide large language model decoding and training

Original

Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. 2023 · 2023

Cited alongside, same era.

Kto: Model alignment as prospect theoretic optimization

Original

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024 · 2024

Closest in time.

Teaching large language models to reason with reinforcement learning

Original

Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. 2024 · 2024

Closest in time.

Training chain-of-thought via latent-variable inference

Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. 2024 · 2024

Closest in time.

Minicpm: Unveiling the potential of small language models with scalable training strategies

Original

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024 · 2024

Closest in time.