2023

The Wisdom of Hindsight Makes Language Models Better Instruction Followers

Zhang, Tianjun, Liu, Fangchen, Wong, Justin et al.

Understand

Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback.

The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models.
However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks.
In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised manner.

Built on

A reduction of imitation learning and structured prediction to no-regret online learning
Ross, S., Gordon, G., and Bagnell, D · 2011
Earlier work this paper cites.
Hindsight experience replay
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W · 2017
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D · 2017
Earlier work this paper cites.
Program induction by rationale generation: Learning to solve and explain algebraic word problems
Original
Ling, W., Yogatama, D., Dyer, C., and Blunsom, P · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
Reward learning from human preferences and demonstrations in atari
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D · 2018
Earlier work this paper cites.
Improving a neural semantic parser by counterfactual learning from human bandit feedback
Original
Lawrence, C. and Riezler, S · 2018
Earlier work this paper cites.
Scalable agent alignment via reward modeling: a research direction
Original
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S · 2018
Earlier work this paper cites.
Multi-goal reinforcement learning: Challenging robotics environments and request for research
Original
Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al · 2018
Earlier work this paper cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S · 2018
Earlier work this paper cites.
Hg-dagger: Interactive imitation learning with human experts
Kelly, M., Sidrane, C., Driggs-Campbell, K., and Kochenderfer, M. J · 2019
Earlier work this paper cites.
Superglue: A stickier benchmark for general-purpose language understanding systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S · 2019
Earlier work this paper cites.
Fine-tuning language models from human preferences
Original
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G · 2019
Earlier work this paper cites.
Making pre-trained language models better few-shot learners
Original
Gao, T., Fisch, A., and Chen, D · 2020
Earlier work this paper cites.
Conservative q-learning for offline reinforcement learning
Kumar, A., Zhou, A., Tucker, G., and Levine, S · 2020
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al · 2020
Earlier work this paper cites.

Similar

Autoprompt: Eliciting knowledge from language models with automatically generated prompts
Original
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S · 2020
Cited alongside, same era.
Learning to summarize with human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F · 2020
Cited alongside, same era.
A general language assistant as a laboratory for alignment
Original
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al · 2021
Cited alongside, same era.
On the dangers of stochastic parrots: Can language models be too big?
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S · 2021
Cited alongside, same era.
On the opportunities and risks of foundation models
Original
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al · 2021
Cited alongside, same era.
Training verifiers to solve math word problems
Original
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al · 2021
Cited alongside, same era.
Measuring mathematical problem solving with the math dataset
Original
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J · 2021
Cited alongside, same era.
Offline reinforcement learning as one big sequence modeling problem
Janner, M., Li, Q., and Levine, S · 2021
Cited alongside, same era.
The power of scale for parameter-efficient prompt tuning
Original
Lester, B., Al-Rfou, R., and Constant, N · 2021
Cited alongside, same era.
Webgpt: Browser-assisted question-answering with human feedback
Original
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al · 2021
Cited alongside, same era.
True few-shot learning with language models
Perez, E., Kiela, D., and Cho, K · 2021
Cited alongside, same era.
Ethical and social risks of harm from language models
Original
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al · 2021
Cited alongside, same era.

Then

Contrastive learning as goal-conditioned reinforcement learning
Original
Eysenbach, B., Zhang, T., Salakhutdinov, R., and Levine, S · 2022
Later among the works it cites.
Large language models can self-improve
Original
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J · 2022
Later among the works it cites.
Efficient planning in a compact latent action space
Original
Jiang, Z., Zhang, T., Janner, M., Li, Y., Rocktäschel, T., Grefenstette, E., and Tian, Y · 2022
Later among the works it cites.
Large language models are zero-shot reasoners
Original
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y · 2022
Later among the works it cites.
In-context reinforcement learning with algorithm distillation
Original
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al · 2022
Later among the works it cites.
Solving quantitative reasoning problems with language models
Original
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al · 2022
Later among the works it cites.
Chatgpt: Optimizing language models for dialogue, Nov 2022
OpenAI · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al · 2022
Later among the works it cites.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models
Original
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al · 2022
Later among the works it cites.
Solving math word problems with process-and outcome-based feedback
Original
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I · 2022
Later among the works it cites.
Chain of thought prompting elicits reasoning in large language models
Original
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D · 2022
Later among the works it cites.
Star: Self-taught reasoner bootstrapping reasoning with reasoning
Zelikman, E., Mu, J., Goodman, N. D., and Wu, Y. T · 2022
Later among the works it cites.
Tempera: Test-time prompting via reinforcement learning
Original
Zhang, T., Wang, X., Zhou, D., Schuurmans, D., and Gonzalez, J. E · 2022
Later among the works it cites.
Online decision transformer
Zheng, Q., Zhang, A., and Grover, A · 2022
Later among the works it cites.
Least-to-most prompting enables complex reasoning in large language models
Original
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., and Chi, E · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…