Minor DPO reject penalty to increase training robustness

Proximal policy optimization algorithms

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017) · 2017

Earlier work this paper cites.

Fine-tuning language models from human preferences

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2020) · 2020

Earlier work this paper cites.

A general language assistant as a laboratory for alignment

Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021) · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021) · 2021

Earlier work this paper cites.

The cringe loss: Learning what language not to model

Adolphs, L., Gao, T., Xu, J., Shuster, K., Sukhbaatar, S., and Weston, J. (2022) · 2022

Earlier work this paper cites.

A simple contrastive learning objective for alleviating neural text degeneration

Jiang, S., Zhang, R., Vakulenko, S., and de Rijke, M. (2022) · 2022

Earlier work this paper cites.

Webgpt: Browser-assisted question-answering with human feedback

Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2022) · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022) · 2022

Cited alongside, same era.

A general theoretical paradigm to understand learning from human preferences

Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. (2023) · 2023

Cited alongside, same era.

Qwen technical report

Original

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. (2023) · 2023

Cited alongside, same era.

Direct preference optimization: Your language model is secretly a reward model

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023) · 2023

Cited alongside, same era.

Orpo: Monolithic preference optimization without reference model

Hong, J., Lee, N., and Thorne, J. (2024) · 2024

Closest in time.

Simpo: Simple preference optimization with a reference-free reward

Meng, Y., Xia, M., and Chen, D. (2024) · 2024

Closest in time.

Smaug: Fixing failure modes of preference optimisation with dpo-positive

Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. (2024) · 2024

Closest in time.

From r r to q ∗ q^{*} : Your language model is secretly a q-function

Rafailov, R., Hejna, J., Park, R., and Finn, C. (2024) · 2024

Closest in time.

Metamath: Bootstrap your own mathematical questions for large language models

Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. (2024) · 2024

Closest in time.

Llamafactory: Unified efficient fine-tuning of 100+ language models