Fetching the paper…

Reading the bibliography…

2024

Transforming and Combining Rewards for Aligning Large Language Models

Wang, Zihao, Nagpal, Chirag, Berant, Jonathan et al.

Original
PDF
Post on X

Understand

A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language model.

We study two closely related problems that arise in this approach.
First, any monotone transformation of the reward model preserves preference ranking; is there a choice that is ``better'' than others? Second, we often wish to align language models to multiple properties: how should we combine multiple reward models? Using a probabilistic interpretation of the alignment procedure, we identify a natural choice for transformation for (the common case of) rewards learned from Bradley-Terry preference models.
The derived transformation is straightforward: we apply a log-sigmoid function to the centered rewards, a method we term ``LSC-transformation'' (log-sigmoid-centered transformation).

Transforming and Combining Rewards for Aligning Large Language Models · Around

Built on

“Rank analysis of incomplete block designs: I. The method of paired comparisons”
Ralph Bradley and Milton Terry · 1952
Earlier work this paper cites.
“Fine-tuning language models from human preferences”
Original
Daniel Ziegler et al · 2019
Earlier work this paper cites.
“Exploring the limits of transfer learning with a unified text-to-text transformer”
Colin Raffel et al · 2020
Earlier work this paper cites.
“Learning to summarize with human feedback”
Nisan Stiennon et al · 2020
Earlier work this paper cites.
“Training a helpful and harmless assistant with reinforcement learning from human feedback”
Original
Yuntao Bai et al · 2022
Earlier work this paper cites.
“Fine-tuning language models to find agreement among humans with diverse preferences”, 2022
Original
Michiel. Bakker et al · 2022
Earlier work this paper cites.
“Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control”
Original
Xiang Fan et al · 2022
Earlier work this paper cites.
“RL with KL penalties is better viewed as Bayesian inference”
Original
Tomasz Korbak, Ethan Perez and Christopher Buckley · 2022
Earlier work this paper cites.

Similar

“Training language models to follow instructions with human feedback”
Long Ouyang et al · 2022
Cited alongside, same era.
“Calibrating sequence likelihood improves conditional language generation”
Original
Yao Zhao et al · 2022
Cited alongside, same era.
“Palm 2 technical report”
Original
Rohan Anil et al · 2023
Cited alongside, same era.
“A general theoretical paradigm to understand learning from human preferences”
Original
Mohammad Azar et al · 2023
Cited alongside, same era.
“Reward model ensembles help mitigate overoptimization”
Original
Thomas Coste, Usman Anwar, Robert Kirk and David Krueger · 2023
Cited alongside, same era.

Then

“Large language models are zero-shot rankers for recommender systems”
Original
Yupeng Hou et al · 2023
Later among the works it cites.
“Confronting Reward Model Overoptimization with Constrained RLHF”, 2023
Original
Ted Moskovitz et al · 2023
Later among the works it cites.
“Direct preference optimization: Your language model is secretly a reward model”
Original
Rafael Rafailov et al · 2023
Later among the works it cites.
“The trickle-down impact of reward (in-) consistency on rlhf”
Original
Lingfeng Shen et al · 2023
Later among the works it cites.
“A long way to go: Investigating length correlations in rlhf”
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

“Alpacafarm: A simulation framework for methods that learn from human feedback”

Yann Dubois et al · 2023

Cited alongside, same era.

“Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking”

Jacob Eisenstein et al · 2023

Cited alongside, same era.

“Scaling laws for reward model overoptimization”

Leo Gao, John Schulman and Jacob Hilton · 2023

Cited alongside, same era.

Prasann Singhal, Tanya Goyal, Jiacheng Xu and Greg Durrett · 2023

Later among the works it cites.

“Fine-Grained Human Feedback Gives Better Rewards for Language Model Training”, 2023

Zeqiu Wu et al · 2023

Later among the works it cites.

“Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles”

Yuanzhao Zhai et al · 2023

Later among the works it cites.

“WARM: On the Benefits of Weight Averaged Reward Models”

Alexandre Ram\’e et al · 2024

Closest in time.