2023

Pretraining Language Models with Human Preferences

Korbak, Tomasz, Shi, Kejian, Chen, Angelica et al.

Understand

Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more.

Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences.
We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs.
We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model.

Built on

Roberta: A robustly optimized bert pretraining approach, 2019
Original
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V · 1907
Earlier work this paper cites.
Ctrl: A conditional transformer language model for controllable generation, 2019
Original
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R · 1909
Earlier work this paper cites.
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019
Original
Peng, X. B., Kumar, A., Zhang, G., and Levine, S · 1910
Earlier work this paper cites.
Reward-conditioned policies, 2019
Original
Kumar, A., Peng, X. B., and Levine, S · 1912
Earlier work this paper cites.
Reinforcement learning upside down: Don’t predict rewards – just map them to actions, 2019
Original
Schmidhuber, J · 1912
Earlier work this paper cites.
SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L · 2001
Earlier work this paper cites.
Style guide for Python code
van Rossum, G., Warsaw, B., and Coghlan, N · 2001
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C · 2003
Earlier work this paper cites.
The pascal recognising textual entailment challenge
Dagan, I., Glickman, O., and Magnini, B · 2005
Earlier work this paper cites.
Automatically constructing a corpus of sentential paraphrases
Dolan, W. B. and Brockett, C · 2005
Earlier work this paper cites.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020
Original
Levine, S., Kumar, A., Tucker, G., and Fu, J · 2005
Earlier work this paper cites.
The second pascal recognising textual entailment challenge
Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., and Giampiccolo, D · 2006
Earlier work this paper cites.
Awac: Accelerating online reinforcement learning with offline datasets, 2020
Original
Nair, A., Gupta, A., Dalal, M., and Levine, S · 2006
Earlier work this paper cites.
The third PASCAL recognizing textual entailment challenge
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B · 2007
Earlier work this paper cites.
Reinforcement learning by reward-weighted regression for operational space control
Peters, J. and Schaal, S · 2007
Earlier work this paper cites.
The fifth PASCAL recognizing textual entailment challenge
Bentivogli, L., Magnini, B., Dagan, I., Dang, H. T., and Giampiccolo, D · 2009
Earlier work this paper cites.
Recipes for safety in open-domain chatbots, 2020
Original
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E · 2010
Earlier work this paper cites.
The winograd schema challenge
Levesque, H. J · 2011
Earlier work this paper cites.
Extracting training data from large language models, 2020
Original
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C · 2012
Earlier work this paper cites.
Context dependent recurrent neural network language model
Mikolov, T. and Zweig, G · 2012
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C · 2013
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B · 2016
Earlier work this paper cites.
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R · 2016
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P · 2016
Earlier work this paper cites.
Controlling linguistic style aspects in neural language generation
Ficler, J. and Goldberg, Y · 2017
Earlier work this paper cites.
Hierarchical neural story generation
Fan, A., Lewis, M., and Dauphin, Y · 2018
Earlier work this paper cites.
Ethical challenges in data-driven dialogue systems
Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J · 2018
Earlier work this paper cites.
Towards controllable story generation
Peng, N., Ghazvininejad, M., May, J., and Knight, K · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Radford, A. and Narasimhan, K · 2018
Earlier work this paper cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S · 2018
Earlier work this paper cites.

Similar

Neural network acceptability judgments
Original
Warstadt, A., Singh, A., and Bowman, S. R · 2018
Cited alongside, same era.
A broad-coverage challenge corpus for sentence understanding through inference
Williams, A., Nangia, N., and Bowman, S · 2018
Cited alongside, same era.
Texygen: A benchmarking platform for text generation models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y · 2018
Cited alongside, same era.
Nuanced metrics for measuring unintended bias with real data for text classification
Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L · 2019
Cited alongside, same era.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Carlini, N., Liu, C., Erlingsson, U., Kos, J., and Song, D · 2019
Cited alongside, same era.
Style transformer: Unpaired text style transfer without disentangled latent representation
Dai, N., Liang, J., Qiu, X., and Huang, X · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
Using self-supervised learning can improve model robustness and uncertainty
Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019
Cited alongside, same era.
The risk of racial bias in hate speech detection
Sap, M., Card, D., Gabriel, S., Choi, Y., and Smith, N. A · 2019
Cited alongside, same era.
Distilling task-specific knowledge from bert into simple neural networks
Original
Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. J · 2019
Cited alongside, same era.
Fine-tuning language models from human preferences
Original
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G · 2019
Cited alongside, same era.

Then

Challenges in detoxifying language models
Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S · 2021
Later among the works it cites.
Detoxifying language models risks marginalizing minority voices
Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D · 2021
Later among the works it cites.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J · 2022
Later among the works it cites.
Quantifying memorization across neural language models, 2022
Original
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C · 2022
Later among the works it cites.
Scaling instruction-finetuned language models, 2022
Original
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J · 2022
Later among the works it cites.
Rvs: What is essential for offline RL via supervised learning?
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S · 2022
Later among the works it cites.
An empirical analysis of compute-optimal large language model training
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L · 2022
Later among the works it cites.
GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems
Jang, Y., Lee, J., and Kim, K.-E · 2022
Later among the works it cites.
RL with KL penalties is better viewed as Bayesian inference
Korbak, T., Perez, E., and Buckley, C · 2022
Later among the works it cites.
TruthfulQA: Measuring how models mimic human falsehoods
Lin, S., Hilton, J., and Evans, O · 2022
Later among the works it cites.
QUARK: Controllable text generation with reinforced unlearning
Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y · 2022
Later among the works it cites.
Teaching language models to support answers with verified quotes, 2022
Original
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R · 2022
Later among the works it cites.
Red teaming language models with language models, 2022
Original
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G · 2022
Later among the works it cites.
Effect of scale on catastrophic forgetting in neural networks
Ramasesh, V. V., Lewkowycz, A., and Dyer, E · 2022
Later among the works it cites.
Multitask prompted training enables zero-shot task generalization
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, A. M · 2022
Later among the works it cites.
Training language models with language feedback, 2022
Original
Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E · 2022
Later among the works it cites.
Offline rl for natural language generation with implicit language q learning, 2022
Original
Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S · 2022
Later among the works it cites.
Transcending scaling laws with 0.1
Original
Tay, Y., Wei, J., Chung, H. W., Tran, V. Q., So, D. R., Shakeri, S., Garcia, X., Zheng, H. S., Rao, J., Chowdhery, A., Zhou, D., Metzler, D., Petrov, S., Houlsby, N., Le, Q. V., and Dehghani, M · 2022
Later among the works it cites.
Natural Language Processing with Transformers: Building Language Applications with Hugging Face
Tunstall, L., von Werra, L., and Wolf, T · 2022
Later among the works it cites.
Will we run out of data? an analysis of the limits of scaling datasets in machine learning, 2022
Original
Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A · 2022
Later among the works it cites.
Overcoming catastrophic forgetting in zero-shot cross-lingual generation
Vu, T., Barua, A., Lester, B., Cer, D., Iyyer, M., and Constant, N · 2022
Later among the works it cites.
Exploring the limits of domain-adaptive training for detoxifying large-scale language models
Wang, B., Ping, W., Xiao, C., Xu, P., Patwary, M., Shoeybi, M., Li, B., Anandkumar, A., and Catanzaro, B · 2022
Later among the works it cites.
Adversarial training for high-stakes reliability
Ziegler, D., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., de Haas, D., Shlegeris, B., and Thomas, N · 2022
Later among the works it cites.
Palm 2 technical report, 2023
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbury, J., Brahma, S., Brooks, K., Catasta, M., Cheng, Y., Cherry, C., Choquette-Choo, C. A., Chowdhery, A., Crepy, C., Dave, S., Dehghani, M., Dev, S., Devlin, J., Díaz, M., Du, N., Dyer, E., Feinberg, V., Feng, F., Fienber, V., Freitag, M., Garcia, X., Gehrmann, S., Gonzalez, L., Gur-Ari, G., Hand, S., Hashemi, H., Hou, L., Howland, J., Hu, A., Hui, J., Hurwitz, J., Isard, M., Ittycheriah, A., Jagielski, M., Jia, W., Kenealy, K., Krikun, M., Kudugunta, S., Lan, C., Lee, K., Lee, B., Li, E., Li, M., Li, W., Li, Y., Li, J., Lim, H., Lin, H., Liu, Z., Liu, F., Maggioni, M., Mahendru, A., Maynez, J., Misra, V., Moussalem, M., Nado, Z., Nham, J., Ni, E., Nystrom, A., Parrish, A., Pellat, M., Polacek, M., Polozov, A., Pope, R., Qiao, S., Reif, E., Richter, B., Riley, P., Ros, A. C., Roy, A., Saeta, B., Samuel, R., Shelby, R., Slone, A., Smilkov, D., So, D. R., Sohn, D., Tokumine, S., Valter, D., Vasudevan, V., Vodrahalli, K., Wang, X., Wang, P., Wang, Z., Wang, T., Wieting, J., Wu, Y., Xu, K., Xu, Y., Xue, L., Yin, P., Yu, J., Zhang, Q., Zheng, S., Zheng, C., Zhou, W., Zhou, D., Petrov, S., and Wu, Y · 2023
Closest in time.
Improving code generation by training with natural language feedback, 2023
Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E · 2023
Closest in time.
The case for 4-bit precision: k-bit inference scaling laws, 2023
Dettmers, T. and Zettlemoyer, L · 2023
Closest in time.
Aligning language models with preferences through f-divergence minimization, 2023
Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M · 2023
Closest in time.
Training language models with language feedback at scale, 2023
Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen, A., Cho, K., and Perez, E · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…