2024

Self-Consistency Preference Optimization

Prasad, Archiki, Yuan, Weizhe, Pang, Richard Yuanzhe et al.

Understand

Self-alignment, whereby models learn to improve themselves without human annotation, is a rapidly growing research area.

However, existing techniques often fail to improve complex reasoning tasks due to the difficulty of assigning correct rewards.
An orthogonal approach that is known to improve correctness is self-consistency, a method applied at inference time based on multiple sampling in order to find the most consistent answer.
In this work, we extend the self-consistency concept to help train models.

Built on

A new asymmetric measure of association for ordinal variables
Somers, R. H · 1962
Earlier work this paper cites.
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
Fischler, M. A. and Bolles, R. C · 1981
Earlier work this paper cites.
Hybrid algorithms for the constraint satisfaction problem
Prosser, P · 1993
Earlier work this paper cites.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y. and Ghahramani, Z · 2016
Earlier work this paper cites.
Execution-guided neural program synthesis
Chen, X., Liu, C., and Song, D · 2018
Earlier work this paper cites.
Addressing failure prediction by learning model confidence
Corbière, C., Thome, N., Bar-Hen, A., Cord, M., and Pérez, P · 2019
Earlier work this paper cites.
Improving model calibration with accuracy versus uncertainty optimization
Krishnan, R. and Tickoo, O · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al · 2021
Earlier work this paper cites.
Measuring mathematical problem solving with the math dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J · 2021
Earlier work this paper cites.
Ethical and social risks of harm from language models
Original
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al · 2021
Earlier work this paper cites.
Training a helpful and harmless assistant with reinforcement learning from human feedback
Original
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al · 2022
Earlier work this paper cites.
Large language models are zero-shot reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y · 2022
Earlier work this paper cites.
Competition-level code generation with alphacode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al · 2022
Earlier work this paper cites.
Natural language to code translation with execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., and Wang, S. I · 2022
Earlier work this paper cites.
Solving math word problems with process-and outcome-based feedback
Original
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al · 2022
Earlier work this paper cites.

Similar

STaR: Bootstrapping reasoning with reasoning
Zelikman, E., Wu, Y., Mu, J., and Goodman, N · 2022
Cited alongside, same era.
Open problems and fundamental limitations of reinforcement learning from human feedback
Original
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al · 2023
Cited alongside, same era.
Codet: Code generation with generated tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W · 2023
Cited alongside, same era.
RAFT: Reward ranked finetuning for generative foundation model alignment
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T · 2023
Cited alongside, same era.
Large language models can self-improve
Huang, J., Gu, S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J · 2023
Cited alongside, same era.
Llama 2: Open foundation and fine-tuned chat models, 2023
Original
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T · 2023
Cited alongside, same era.
Iterative DPO alignment
Tran, H., Glaze, C., and Hancock, B · 2023
Cited alongside, same era.
Self-consistency improves chain of thought reasoning in language models
Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D · 2023
Cited alongside, same era.
Some things are more cringe than others: Preference optimization with the pairwise cringe loss
Original
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J · 2023
Cited alongside, same era.
The claude 3 model family: Opus, sonnet, haiku
Anthropic · 2024
Cited alongside, same era.
Universal self-consistency for large language models
Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang, X., and Zhou, D · 2024
Cited alongside, same era.
The llama 3 herd of models
Original
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al · 2024
Cited alongside, same era.

Then

Rewardbench: Evaluating reward models for language modeling
Original
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al · 2024
Closest in time.
Internal consistency and self-feedback in large language models: A survey
Original
Liang, X., Song, S., Zheng, Z., Wang, H., Yu, Q., Li, X., Li, R.-H., Xiong, F., and Li, Z · 2024
Closest in time.
Let’s verify step by step
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K · 2024
Closest in time.
Generative reward models
Original
Mahan, D., Van Phung, D., Rafailov, R., Blagden, C., Lile, N., Castricato, L., Fränken, J.-P., Finn, C., and Albalak, A · 2024
Closest in time.
Iterative reasoning preference optimization
Pang, R. Y., Yuan, W., He, H., Cho, K., Sukhbaatar, S., and Weston, J · 2024
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C · 2024
Closest in time.
Direct nash optimization: Teaching language models to self-improve with general preferences
Original
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadallah, A., and Xie, T · 2024
Closest in time.
Mapo: Advancing multilingual reasoning through multilingual-alignment-as-preference optimization
She, S., Zou, W., Huang, S., Zhu, W., Liu, X., Geng, X., and Chen, J · 2024
Closest in time.
Beyond human data: Scaling self-training for problem-solving with language models
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G. F., Sedghi, H., Mordatch, I., Simpson, I., Gur, I., Snoek, J., Pennington, J., Hron, J., Kenealy, K., Swersky, K., Mahajan, K., Culp, L. A., Xiao, L., Bileschi, M., Constant, N., Novak, R., Liu, R., Warkentin, T., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel, N · 2024
Closest in time.
Gemma 2: Improving open language models at a practical size
Original
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., et al · 2024
Closest in time.
Interpretable preferences via multi-objective reward modeling and mixture-of-experts
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T · 2024
Closest in time.
Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., LI, Y., Fu, J., He, J., and Hooi, B · 2024
Closest in time.
MetaMath: Bootstrap your own mathematical questions for large language models
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W · 2024
Closest in time.
Self-rewarding language models
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E · 2024
Closest in time.
Generative verifiers: Reward modeling as next-token prediction
Original
Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A., and Agarwal, R · 2024
Closest in time.
Preference optimization for reasoning with pseudo feedback
Jiao, F., Guo, G., Zhang, X., Chen, N. F., Joty, S., and Wei, F · 2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…