Selective Preference Optimization via Token-Level Reward Function Estimation

Rank analysis of incomplete block designs: I. the method of paired comparisons

Ralph Allan Bradley and Milton E Terry. 1952 · 1952

Earlier work this paper cites.

Maximum entropy inverse reinforcement learning

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. 2008 · 2008

Earlier work this paper cites.

Proximal policy optimization algorithms

Original

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 · 2017

Earlier work this paper cites.

Learning to summarize with human feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020 · 2020

Earlier work this paper cites.

Training a helpful and harmless assistant with reinforcement learning from human feedback

Original

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Pythia: A suite for analyzing large language models across training and scaling

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023 · 2023

Earlier work this paper cites.

Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

Original

Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. 2023 · 2023

Earlier work this paper cites.

Ultrafeedback: Boosting language models with high-quality feedback

Original

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023 · 2023

Earlier work this paper cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao. 2023 · 2023

Earlier work this paper cites.

Enhancing chat language models by scaling high-quality instructional conversations

Original

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023 · 2023

Earlier work this paper cites.

Scaling laws for reward model overoptimization

Leo Gao, John Schulman, and Jacob Hilton. 2023 · 2023

Earlier work this paper cites.

Ai alignment: A comprehensive survey

Original

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023 · 2023

Earlier work this paper cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Cited alongside, same era.

Rrhf: Rank responses to align language models with human feedback without tears

Original

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023 · 2023

Cited alongside, same era.

A general theoretical paradigm to understand learning from human preferences

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024 · 2024

Cited alongside, same era.

Dense reward for free in reinforcement learning from human feedback

Original

Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. 2024 · 2024

Cited alongside, same era.

Quantifying the gain in weak-to-strong generalization

Rho-1: Not all tokens are what you need

Original

Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. 2024 · 2024

Closest in time.

Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence

Original

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024 · 2024

Closest in time.

Simpo: Simple preference optimization with a reference-free reward

Original

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024 · 2024

Closest in time.

Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold

Original

Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. 2024 · 2024

Closest in time.