Discovering Preference Optimization Algorithms with and for Large Language Models

Rank analysis of incomplete block designs: I. the method of paired comparisons

Ralph Allan Bradley and Milton E Terry · 1952

Earlier work this paper cites.

Temporal credit assignment in reinforcement learning

Richard Stuart Sutton · 1984

Earlier work this paper cites.

A training algorithm for optimal margin classifiers

Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik · 1992

Earlier work this paper cites.

Support-vector networks

Corinna Cortes and Vladimir Vapnik · 1995

Earlier work this paper cites.

Are loss functions all the same?

Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri · 2004

Earlier work this paper cites.

Maximum entropy inverse reinforcement learning

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al · 2008

Earlier work this paper cites.

Learning word vectors for sentiment analysis

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts · 2011

Earlier work this paper cites.

Deep reinforcement learning from human preferences

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017

Earlier work this paper cites.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter · 2017

Earlier work this paper cites.

Automatic differentiation in pytorch

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer · 2017

Earlier work this paper cites.

Tl; dr: Mining reddit to learn automatic summarization

Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein · 2017

Earlier work this paper cites.

Evolved policy gradients

Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel · 2018

Earlier work this paper cites.

Implementation matters in deep rl: A case study on ppo and trpo

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry · 2019

Earlier work this paper cites.

Way off-policy batch deep reinforcement learning of implicit human preferences in dialog

Original

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard · 2019

Earlier work this paper cites.

Improving generalization in meta reinforcement learning using learned objectives

Original

Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019

Earlier work this paper cites.

Meta-learning curiosity algorithms

Original

Ferran Alet, Martin F Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling · 2020

Earlier work this paper cites.

Realtoxicityprompts: Evaluating neural toxic degeneration in language models

Original

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith · 2020

Earlier work this paper cites.

Discovering reinforcement learning algorithms

Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver · 2020

Earlier work this paper cites.

Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients

Brenden K Petersen, Mikel Landajuela Larma, Terrell N Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim · 2020

Earlier work this paper cites.

Learning to summarize with human feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano · 2020

Earlier work this paper cites.

Neural symbolic regression that scales

Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo · 2021

Earlier work this paper cites.

Extracting training data from large language models

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al · 2021

Earlier work this paper cites.

Evolving reinforcement learning algorithms

Original

John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V Le, Honglak Lee, and Aleksandra Faust · 2021

Cited alongside, same era.

Meta-learning in neural networks: A survey

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey · 2021

Cited alongside, same era.

Symbolic regression via neural-guided genetic programming population seeding

T Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P Santiago, Daniel M Faissol, and Brenden K Petersen · 2021

Cited alongside, same era.

Neural laplace: Learning diverse classes of differential equations in the laplace domain

Samuel I Holt, Zhaozhi Qian, and Mihaela van der Schaar · 2022

Cited alongside, same era.

Discovered policy optimisation

Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster · 2022

Cited alongside, same era.

Some things are more cringe than others: Preference optimization with the pairwise cringe loss

Original

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston · 2023

Later among the works it cites.

Language to rewards for robotic skill synthesis

Original

Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al · 2023

Later among the works it cites.

Slic-hf: Sequence likelihood calibration with human feedback

Original

Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu · 2023

Later among the works it cites.

Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

Original

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker · 2024

Closest in time.