Fetching the paper…

Reading the bibliography…

2024

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

Hao, Shugang, Duan, Lingjie

Original
PDF
Post on X

Understand

Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences.

However, human labelers are selfish and have diverse preferences.
They may strategically misreport their online feedback to influence the system's aggregation towards their own preferences.
Current practice simply averages labelers' feedback per time and fails to identify the most accurate human labeler, leading to linear regret $\mathcal{O}(T)$ for $T$ time slots.

Online Learning from Strategic Human Feedback in LLM Fine-Tuning · Around

Built on

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
T. Roughgarden and O. Schrijvers, “Online prediction with selfish experts,” Advances in Neural Information Processing Systems , vol. 30, 2017
2017
Earlier work this paper cites.
M. Völske, M. Potthast, S. Syed, and B. Stein, “Tl; dr: Mining reddit to learn automatic summarization,” in Proceedings of the Workshop on New Frontiers in Summarization , 2017, pp. 59–63
2017
Earlier work this paper cites.
R. Freeman, D. Pennock, C. Podimata, and J. W. Vaughan, “No-regret and incentive-compatible online learning,” in International Conference on Machine Learning . PMLR, 2020, pp. 3270–3279
2020
Earlier work this paper cites.
A. Pacchiano, A. Saha, and J. Lee, “Dueling rl: reinforcement learning with trajectory preferences,” arXiv preprint arXiv:2111.04850 , 2021
Original
2021
Earlier work this paper cites.
R. Frongillo, R. Gomez, A. Thilagar, and B. Waggoner, “Efficient competitions and online learning with strategic forecasters,” in Proceedings of the 22nd ACM Conference on Economics and Computation , 2021, pp. 479–496
2021
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022
2022
Earlier work this paper cites.
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al. , “Training a helpful and harmless assistant with reinforcement learning from human feedback. corr, abs/2204.05862, 2022a. doi: 10.48550,” arXiv preprint arXiv.2204.05862 , 2022
Original
2022
Earlier work this paper cites.
M. Asadi, A. Bellet, O.-A. Maillard, and M. Tommasi, “Collaborative algorithms for online personalized mean estimation,” Transactions on Machine Learning Research Journal , 2022
2022
Earlier work this paper cites.
X. Chen, H. Zhong, Z. Yang, Z. Wang, and L. Wang, “Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation,” in International Conference on Machine Learning . PMLR, 2022, pp. 3773–3793
2022
Earlier work this paper cites.

Similar

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Open and efficient foundation language models,” Preprint at arXiv. https://doi. org/10.48550/arXiv , vol. 2302, 2023
2023
Cited alongside, same era.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023
Original
2023
Cited alongside, same era.
A. Havrilla, M. Zhuravinskyi, D. Phung, A. Tiwari, J. Tow, S. Biderman, Q. Anthony, and L. Castricato, “trlx: A framework for large scale reinforcement learning from human feedback,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023, pp. 8578–8595
2023
Cited alongside, same era.
A. Köpf, Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi et al. , “Openassistant conversations-democratizing large language model alignment,” Advances in Neural Information Processing Systems , vol. 36, 2024

Then

S. Cen, J. Mei, K. Goshvadi, H. Dai, T. Yang, S. Yang, D. Schuurmans, Y. Chi, and B. Dai, “Value-incentivized preference optimization: A unified approach to online and offline rlhf,” arXiv preprint arXiv:2405.19320 , 2024
Original
2024
Closest in time.
S. Zhang, D. Yu, H. Sharma, Z. Yang, S. Wang, H. Hassan, and Z. Wang, “Self-exploring language models: Active preference elicitation for online alignment,” arXiv preprint arXiv:2405.19332 , 2024
Original
2024
Closest in time.
L. Chen, J. Chen, C. Liu, J. Kirchenbauer, D. Soselia, C. Zhu, T. Goldstein, T. Zhou, and H. Huang, “Optune: Efficient online preference tuning,” arXiv preprint arXiv:2406.07657 , 2024
Original
2024
Closest in time.
C. Park, M. Liu, D. Kong, K. Zhang, and A. E. Ozdaglar, “Rlhf from heterogeneous feedback via personalization and preference aggregation,” in ICML 2024 Workshop on Theoretical Foundations of Foundation Models , 2024
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2024

Cited alongside, same era.

H. Sun, Y. Chen, S. Wang, W. Chen, and X. Deng, “Mechanism design for llm fine-tuning with multiple reward models,” arXiv preprint arXiv:2405.16276 , 2024

2024

Cited alongside, same era.

W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang, “Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint,” in Forty-first International Conference on Machine Learning , 2024

2024

Cited alongside, same era.

E. Soumalias, M. J. Curry, and S. Seuken, “Truthful aggregation of llms with an application to online advertising,” arXiv preprint arXiv:2405.05905 , 2024

2024

Cited alongside, same era.

V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf et al. , “Social choice for ai alignment: Dealing with diverse human feedback,” arXiv preprint arXiv:2404.10271 , 2024

2024

Cited alongside, same era.

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Cited alongside, same era.

T. Xie, D. J. Foster, A. Krishnamurthy, C. Rosset, A. Awadallah, and A. Rakhlin, “Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf,” arXiv preprint arXiv:2405.21046 , 2024

2024

Cited alongside, same era.

K. A. Dubey, Z. Feng, R. Kidambi, A. Mehta, and D. Wang, “Auctions with llm summaries,” arXiv preprint arXiv:2404.08126 , 2024

2024

Closest in time.

Y. Chen, J. Zhu, and K. Kandasamy, “Mechanism design for collaborative normal mean estimation,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

J. Li, M. Li, and H. Chan, “Strategyproof mechanisms for group-fair obnoxious facility location problems,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 9, 2024, pp. 9832–9839

2024

Closest in time.

Y. Wang, H. Zhou, and M. Li, “Positive intra-group externalities in facility location,” in Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , 2024, pp. 1883–1891

2024

Closest in time.

H. Zhong, G. Feng, W. Xiong, L. Zhao, D. He, J. Bian, and L. Wang, “Dpo meets ppo: Reinforced token optimization for rlhf,” arXiv preprint arXiv:2404.18922 , 2024

2024

Closest in time.