Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning

[author] Ziegler, Daniel MD. M., Stiennon, NisanN., Wu, JeffreyJ., Brown, Tom BT. B., Radford, AlecA., Amodei, DarioD., Christiano, PaulP. and Irving, GeoffreyG. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

Original

1909

Earlier work this paper cites.

[author] Bradley, Ralph AllanR. A. and Terry, Milton EM. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39 324–345

1952

Earlier work this paper cites.

[author] May, Kenneth O.K. O. (1954). Intransitivity, Utility, and the Aggregation of Preference Patterns. Econometrica 22 1

1954

Earlier work this paper cites.

[author] Tversky, AmosA. (1969). Intransitivity of preferences. Psychological Review 76 31-48

1969

Earlier work this paper cites.

[author] Gardner, MM. (1970). Mathematical games, the paradox of the nontransitive dice and the elusive principle of indifference

1970

Earlier work this paper cites.

[author] Levine, SergeyS., Kumar, AviralA., Tucker, GeorgeG. and Fu, JustinJ. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

Original

2005

Earlier work this paper cites.

[author] Tsiatis, Anastasios AA. A. (2006). Semiparametric theory and missing data 4. Springer

2006

Earlier work this paper cites.

Maas, A. L

2011

Earlier work this paper cites.

[author] Michaud, Eric J.E. J., Gleave, AdamA. and Russell, Stuart J.S. J. (2020). Understanding Learned Reward Functions. ArXiv abs/2012.05862

Original

2012

Earlier work this paper cites.

[author] Agranov, MarinaM. and Ortoleva, PietroP. (2015). Stochastic Choice and Preferences for Randomization. Journal of Political Economy 125 40 - 68

2015

Earlier work this paper cites.

[author] Christiano, Paul FP. F., Leike, JanJ., Brown, TomT., Martic, MiljanM., Legg, ShaneS. and Amodei, DarioD. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems 30

2017

Earlier work this paper cites.

[author] Schulman, JohnJ., Wolski, FilipF., Dhariwal, PrafullaP., Radford, AlecA. and Klimov, OlegO. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

Original

2017

Earlier work this paper cites.

[author] Vaswani, AshishA., Shazeer, NoamN., Parmar, NikiN., Uszkoreit, JakobJ., Jones, LlionL., Gomez, Aidan NA. N., Kaiser, ŁukaszŁ. and Polosukhin, IlliaI. (2017). Attention is all you need. Advances in neural information processing systems 30

2017

Earlier work this paper cites.

Völske, M

2017

Earlier work this paper cites.

[author] Chakrabortty, AbhishekA. and Cai, TianxiT. (2018). Efficient and adaptive linear regression in semi-supervised settings. Annals of Statistics

2018

Earlier work this paper cites.

[author] Sutton, Richard SR. S., Barto, Andrew GA. G. et al. (2018). Reinforcement learning: An introduction. MIT press Cambridge

2018

Earlier work this paper cites.

[author] Agarwal, AlekhA., Jiang, NanN., Kakade, Sham MS. M. and Sun, WenW. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep 32 96

2019

Earlier work this paper cites.

[author] Zhang, AnruA., Brown, Lawrence DL. D. and Cai, T TonyT. T. (2019). Semi-supervised Inference: General Theory and Estimation of Means. The Annals of Statistics 47 2538–2566

2019

Earlier work this paper cites.

[author] Milano, SilviaS., Taddeo, MariarosariaM. and and, Luciano FloridiL. F. (2021). Ethical aspects of multi-stakeholder recommendation systems. The Information Society 37 35–45. 10.1080/01972243.2020.1832636

2020

Earlier work this paper cites.

Stiennon, N

2020

Earlier work this paper cites.

[author] von Werra, LeandroL., Belkada, YounesY., Tunstall, LewisL., Beeching, EdwardE., Thrush, TristanT., Lambert, NathanN., Huang, ShengyiS., Rasul, KashifK. and Gallouédec, QuentinQ. (2020). TRL: Transformer Reinforcement Learning. https://github.com/huggingface/trl

2020

Earlier work this paper cites.

[author] Askell, AmandaA., Bai, YuntaoY., Chen, AnnaA., Drain, DawnD., Ganguli, DeepD., Henighan, TomT., Jones, AndyA., Joseph, NicholasN., Mann, BenB., DasSarma, NovaN. et al. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861

Original

2021

Earlier work this paper cites.

[author] Bai, YuntaoY., Jones, AndyA., Ndousse, KamalK., Askell, AmandaA., Chen, AnnaA., DasSarma, NovaN., Drain, DawnD., Fort, StanislavS., Ganguli, DeepD., Henighan, TomT. et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

Original

2022

Earlier work this paper cites.

[author] Bakker, MichielM., Chadwick, MartinM., Sheahan, HannahH., Tessler, MichaelM., Campbell-Gillingham, LucyL., Balaguer, JanJ., McAleese, NatN., Glaese, AmeliaA., Aslanides, JohnJ., Botvinick, MattM. et al. (2022). Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems 35 38176–38189

2022

Earlier work this paper cites.

[author] Hartmann, JochenJ., Heitmann, MarkM., Siebert, ChristianC. and Schamp, ChristinaC. (2023). More than a Feeling: Accuracy and Application of Sentiment Analysis. International Journal of Research in Marketing 40 75-87. https://doi.org/10.1016/j.ijresmar.2022.05.005

2022

Earlier work this paper cites.

[author] Huang, ShengyiS., Dossa, Rousslan Fernand JulienR. F. J., Ye, ChangC., Braga, JeffJ., Chakraborty, DipamD., Mehta, KinalK. and Araújo, João G. M.J. G. M. (2022). CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. Journal of Machine Learning Research 23 1–18

2022

Earlier work this paper cites.

[author] Lindner, DavidD. and El-Assady, MennatallahM. (2022). Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning. ArXiv abs/2206.13316

Original

2022

Cited alongside, same era.

[author] Ouyang, LongL., Wu, JeffreyJ., Jiang, XuX., Almeida, DiogoD., Wainwright, CarrollC., Mishkin, PamelaP., Zhang, ChongC., Agarwal, SandhiniS., Slama, KatarinaK., Ray, AlexA. et al. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems 35 27730–27744

2022

Cited alongside, same era.

[author] Uehara, MasatoshiM., Shi, ChengchunC. and Kallus, NathanN. (2022). A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355

Original

2022

Cited alongside, same era.

[author] Zhang, YuqianY. and Bradic, JelenaJ. (2022). High-dimensional semi-supervised learning: in search of optimal inference of the mean. Biometrika 109 387–403

2022

[author] Lee, Seong JinS. J., Sun, Will WeiW. W. and Liu, YufengY. (2024). Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback. arXiv preprint arXiv:2412.19436

Original

2024

Later among the works it cites.

[author] Liu, PangpangP., Shi, ChengchunC. and Sun, Will WeiW. W. (2024). Dual active learning for reinforcement learning from human feedback. arXiv preprint arXiv:2410.02504

Original

2024

Later among the works it cites.

Mukherjee, S

2024

Later among the works it cites.

Munos, R

2024

Later among the works it cites.

Rafailov, R

2024

Later among the works it cites.

[author] Ramesh, Shyam SundharS. S., Hu, YifanY., Chaimalas, IasonI., Mehta, VirajV., Sessa, Pier GiuseppeP. G., Bou Ammar, HaithamH. and Bogunovic, IlijaI. (2024). Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems 37 37100–37137