From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning

Fine-Tuning Language Models from Human Preferences

Original

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019 · 1909

Earlier work this paper cites.

From novice to expert

Patricia Benner. 1982 · 1982

Earlier work this paper cites.

From novice to expert

Patricia Benner et al · 1984

Earlier work this paper cites.

Efficient training of artificial neural networks for autonomous navigation

Dean A Pomerleau. 1991 · 1991

Earlier work this paper cites.

Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory . 101–103

Stuart Russell. 1998 · 1998

Earlier work this paper cites.

Actor-Critic Algorithms. In Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999] , Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (Eds.). The MIT Press, 1008–1014

Vijay R. Konda and John N. Tsitsiklis. 1999 · 1999

Earlier work this paper cites.

Algorithms for inverse reinforcement learning.. In Icml , Vol. 1. 2

Andrew Y Ng, Stuart Russell, et al · 2000

Earlier work this paper cites.

Apprenticeship learning using linear programming. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 (ACM International Conference Proceeding Series, Vol. 307) , William W. Cohen, Andrew McCallum, and Sam T. Roweis (Eds.). ACM, 1032–1039

Umar Syed, Michael H. Bowling, and Robert E. Schapire. 2008 · 2008

Earlier work this paper cites.

A Survey of Monte Carlo Tree Search Methods

Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. 2012 · 2012

Earlier work this paper cites.

Infinite time horizon maximum causal entropy inverse reinforcement learning. In 53rd IEEE conference on decision and control . IEEE, 4911–4916

Michael Bloem and Nicholas Bambos. 2014 · 2014

Earlier work this paper cites.

Trust Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 (JMLR Workshop and Conference Proceedings, Vol. 37) , Francis R. Bach and David M. Blei (Eds.). JMLR.org, 1889–1897

John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. 2015 · 2015

Earlier work this paper cites.

Generative Adversarial Imitation Learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (Eds.). 4565–4573

Jonathan Ho and Stefano Ermon. 2016 · 2016

Earlier work this paper cites.

Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 4299–4307

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017

Earlier work this paper cites.

Proximal Policy Optimization Algorithms

Original

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 · 2017

Earlier work this paper cites.

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 2369–2380

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018 · 2018

Earlier work this paper cites.

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Proceedings of the 28th International Conference on Computational Linguistics . International Committee on Computational Linguistics, Barcelona, Spain (Online), 6609–6625

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020 · 2020

Earlier work this paper cites.

On faithfulness and factuality in abstractive summarization

Original

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020 · 2020

Earlier work this paper cites.

Learning to summarize with human feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020 · 2020

Earlier work this paper cites.

Detecting hallucinated content in conditional neural sequence generation

Original

Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020 · 2020

Earlier work this paper cites.

A survey of inverse reinforcement learning: Challenges, methods and progress

Saurabh Arora and Prashant Doshi. 2021 · 2021

Cited alongside, same era.

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. 2021 · 2021

Cited alongside, same era.

Training a helpful and harmless assistant with reinforcement learning from human feedback

Original

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al · 2022

Cited alongside, same era.

Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. In International Conference on Machine Learning . PMLR, 2160–2177

Heejong Bong and Alessandro Rinaldo. 2022 · 2022

Cited alongside, same era.

Toran Bruce Richards. 2023

2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023

Later among the works it cites.

The Rise and Potential of Large Language Model Based Agents: A Survey

Original

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. 2023 · 2023

Later among the works it cites.

ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023 · 2023

Later among the works it cites.