Cross-lingual Transfer of Reward Models in Multilingual Alignment

Fine-tuning language models from human preferences

Original

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020 · 1909

Earlier work this paper cites.

Rank analysis of incomplete block designs: I. the method of paired comparisons

Ralph Allan Bradley and Milton E Terry. 1952 · 1952

Earlier work this paper cites.

Deep reinforcement learning from human preferences

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Pre-trained language model representations for language generation

Sergey Edunov, Alexei Baevski, and Michael Auli. 2019 · 2019

Earlier work this paper cites.

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Unsupervised cross-lingual representation learning at scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020

Earlier work this paper cites.

Zero: memory optimizations toward training trillion parameter models

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

Learning to summarize with human feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020 · 2020

Earlier work this paper cites.

Trl: Transformer reinforcement learning

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020 · 2020

Earlier work this paper cites.

Better fine-tuning by reducing representational collapse

Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021 · 2021

Earlier work this paper cites.

mT5: A massively multilingual pre-trained text-to-text transformer

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021

Earlier work this paper cites.

XLM-E: Cross-lingual language model pre-training via ELECTRA

Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022 · 2022

Earlier work this paper cites.

8-bit optimizers via block-wise quantization

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Understanding dataset difficulty with 𝒱 \mathcal{V} -usable information

Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022 · 2022

Earlier work this paper cites.

Accelerate: Training and inference at scale made simple, efficient and adaptable

Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022 · 2022

Earlier work this paper cites.

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023 · 2023

Earlier work this paper cites.

Scaling laws for reward model overoptimization

Leo Gao, John Schulman, and Jacob Hilton. 2023 · 2023

Cited alongside, same era.

Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023 · 2023

Cited alongside, same era.

Mistral 7b

Original

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023 · 2023

Cited alongside, same era.

Alpacaeval: An automatic evaluator of instruction-following models

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023

Cited alongside, same era.

Representation projection invariance mitigates representation collapse

Anastasia Razdaibiedina, Ashish Khetan, Zohar Karnin, Daniel Khashabi, and Vivek Madan. 2023 · 2023

Orpo: Monolithic preference optimization without reference model

Jiwoo Hong, Noah Lee, and James Thorne. 2024 · 2024

Closest in time.

Liger-kernel: Efficient triton kernels for llm training

Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, and Siyu Zhu. 2024 · 2024

Closest in time.

The n+ implementation details of RLHF with PPO: A case study on TL;DR summarization

Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. 2024 · 2024

Closest in time.

Rewardbench: Evaluating reward models for language modeling

Original

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024 · 2024

Closest in time.

Simpo: Simple preference optimization with a reference-free reward

Original

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024 · 2024