2023

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

Li, Ziniu, Xu, Tian, Zhang, Yushun et al.

Understand

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm.

While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens.
To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards.
These properties are not exploited in PPO, making it less suitable for RLHF.

Built on

Rank analysis of incomplete block designs: I. the method of paired comparisons
Bradley, R. A. and Terry, M. E · 1952
Earlier work this paper cites.
Approximate confidence intervals
Bartlett, M · 1953
Earlier work this paper cites.
The analysis of permutations
Plackett, R. L · 1975
Earlier work this paper cites.
Reinforcement-learning connectionist systems
Williams, R. J · 1987
Earlier work this paper cites.
Learning to predict by the methods of temporal differences
Sutton, R · 1988
Earlier work this paper cites.
Reinforcement comparison
Dayan, P · 1991
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J · 1992
Earlier work this paper cites.
Actor-critic algorithms
Konda, V. R. and Tsitsiklis, J. N · 1999
Earlier work this paper cites.
The optimal reward baseline for gradient-based reinforcement learning
Weaver, L. and Tao, N · 2001
Earlier work this paper cites.
Approximately optimal approximate reinforcement learning
Kakade, S. M. and Langford, J · 2002
Earlier work this paper cites.
Variance reduction techniques for gradient estimates in reinforcement learning
Greensmith, E., Bartlett, P. L., and Baxter, J · 2004
Earlier work this paper cites.
Individual choice behavior: A theoretical analysis
Luce, R. D · 2005
Earlier work this paper cites.
Analysis and improvement of policy gradient estimation
Zhao, T., Hachiya, H., Niu, G., and Sugiyama, M · 2011
Earlier work this paper cites.
Handbook of Monte Carlo Methods
Kroese, D. P., Taimre, T., and Botev, Z. I · 2013
Earlier work this paper cites.
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Puterman, M. L · 2014
Earlier work this paper cites.
Deep reinforcement learning for dialogue generation
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., and Gao, J · 2016
Earlier work this paper cites.
Sequence level training with recurrent neural networks
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W · 2016
Earlier work this paper cites.
High-dimensional continuous control using generalized advantage estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P · 2016
Earlier work this paper cites.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al · 2016
Earlier work this paper cites.
Bandit structured prediction for learning from partial feedback in statistical machine translation
Original
Sokolov, A., Riezler, S., and Urvoy, T · 2016
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D · 2017
Earlier work this paper cites.
Bandit structured prediction for neural sequence-to-sequence learning
Original
Kreutzer, J., Sokolov, A., and Riezler, S · 2017
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Reinforcement learning for bandit neural machine translation with simulated human feedback
Original
Nguyen, K., Daumé III, H., and Boyd-Graber, J · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
Principled reinforcement learning with human feedback from pairwise or k-wise comparisons
Zhu, B., Jordan, M. I., and Jiao, J · 2017
Earlier work this paper cites.
Optimization methods for large-scale machine learning
Bottou, L., Curtis, F. E., and Nocedal, J · 2018
Earlier work this paper cites.

Similar

Stochastic variance-reduced policy gradient
Papini, M., Binaghi, D., Canonaco, G., Pirotta, M., and Restelli, M · 2018
Cited alongside, same era.
Reinforcement Learning: An Introduction
Sutton, R. S. and Barto, A. G · 2018
Cited alongside, same era.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019
Cited alongside, same era.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Original
Sanh, V., Debut, L., Chaumond, J., and Wolf, T · 2019
Cited alongside, same era.
Low-memory neural network training: A technical report
Original
Sohoni, N. S., Aberger, C. R., Leszczynski, M., Zhang, J., and Ré, C · 2019
Cited alongside, same era.
Grandmaster level in starcraft ii using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al · 2019
Cited alongside, same era.
Optimality and approximation with policy gradient methods in markov decision processes
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Provably efficient exploration in policy optimization
Cai, Q., Yang, Z., Jin, C., and Wang, Z · 2020
Cited alongside, same era.
Implementation matters in deep policy gradients: A case study on ppo and trpo
Original
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A · 2020
Cited alongside, same era.
Bandit Algorithms
Lattimore, T. and Szepesvári, C · 2020
Cited alongside, same era.
Zero: Memory optimizations toward training trillion parameter models
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y · 2020
Cited alongside, same era.

Then

Open problems and fundamental limitations of reinforcement learning from human feedback
Original
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al · 2023
Closest in time.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al · 2023
Closest in time.
Ultrafeedback: Boosting language models with high-quality feedback
Original
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M · 2023
Closest in time.
Raft: Reward ranked finetuning for generative foundation model alignment
Original
Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T · 2023
Closest in time.
Alpacafarm: A simulation framework for methods that learn from human feedback
Original
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Closest in time.
Scaling laws for reward model overoptimization
Gao, L., Schulman, J., and Hilton, J · 2023
Closest in time.
Mistral 7b
Original
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al · 2023
Closest in time.
Understanding the effects of rlhf on llm generalisation and diversity
Original
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R · 2023
Closest in time.
Efficient memory management for large language model serving with pagedattention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I · 2023
Closest in time.
Rlaif: Scaling reinforcement learning from human feedback with ai feedback
Original
Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A · 2023
Closest in time.
Statistical rejection sampling improves preference optimization
Original
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J · 2023
Closest in time.
The flan collection: Designing data and methods for effective instruction tuning
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., and Roberts, A · 2023
Closest in time.
Full parameter fine-tuning for large language models with limited resources
Original
Lv, K., Yang, Y., Liu, T., Gao, Q., Guo, Q., and Qiu, X · 2023
Closest in time.
Nash learning from human feedback
Original
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al · 2023
Closest in time.
Gpt-4 technical report
Original
OpenAI · 2023
Closest in time.
Reward gaming in conditional text generation
Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A., and He, H · 2023
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Original
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C · 2023
Closest in time.
Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization
Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y · 2023
Closest in time.
Efficient rlhf: Reducing the memory usage of ppo
Original
Santacroce, M., Lu, Y., Yu, H., Li, Y., and Shen, Y · 2023
Closest in time.
Reward collapse in aligning large language models
Original
Song, Z., Cai, T., Lee, J. D., and Su, W. J · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models
Original
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al · 2023
Closest in time.
Zephyr: Direct distillation of lm alignment
Original
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al · 2023
Closest in time.
Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf
Original
Xiong, W., Dong, H., Ye, C., Zhong, H., Jiang, N., and Zhang, T · 2023
Closest in time.
Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales
Original
Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A. A., Rasley, J., Zhang, M., Li, C., Holmes, C., et al · 2023
Closest in time.
Rrhf: Rank responses to align language models with human feedback without tears
Original
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F · 2023
Closest in time.
Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms
Original
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., and da Silva, B. C · 2024
Closest in time.
Break the sequential dependency of llm inference using lookahead decoding
Original
Fu, Y., Bailis, P., Stoica, I., and Zhang, H · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…