When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

Sequencer: Sequence-to-sequence learning for end-to-end program repair

Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2021 · 1959

Earlier work this paper cites.

The CoNLL-2014 shared task on grammatical error correction

Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014 · 2014

Earlier work this paper cites.

Deepfix: Fixing common c language errors by deep learning

Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017 · 2017

Earlier work this paper cites.

Getafix: learning to fix bugs automatically

Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. 2019 · 2019

Earlier work this paper cites.

Corpora generation for grammatical error correction

Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019 · 2019

Earlier work this paper cites.

Deepdelta: learning to repair compilation errors

Ali Mesbah, Andrew Rice, Emily Johnston, Nick Glorioso, and Edward Aftandilian. 2019 · 2019

Earlier work this paper cites.

Language (technology) is power: A critical survey of “bias” in NLP

Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020 · 2020

Earlier work this paper cites.

Factual error correction for abstractive summarization models

Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020 · 2020

Earlier work this paper cites.

FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization

Esin Durmus, He He, and Mona Diab. 2020 · 2020

Earlier work this paper cites.

Automatic fact-guided sentence modification

Darsh Shah, Tal Schuster, and Regina Barzilay. 2020 · 2020

Earlier work this paper cites.

Asking and answering questions to evaluate the factual consistency of summaries

Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020 · 2020

Earlier work this paper cites.

Graph-based, self-supervised program repair from diagnostic feedback

Michihiro Yasunaga and Percy Liang. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021 · 2021

Earlier work this paper cites.

QuestEval: Summarization asks for fact-based evaluation

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021 · 2021

Earlier work this paper cites.

Generate & rank: A multi-task framework for math word problems

Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021 · 2021

Earlier work this paper cites.

Evidence-based factual error correction

James Thorne and Andreas Vlachos. 2021 · 2021

Earlier work this paper cites.

Break-it-fix-it: Unsupervised learning for program repair

Michihiro Yasunaga and Percy Liang. 2021 · 2021

Earlier work this paper cites.

Constitutional ai: Harmlessness from ai feedback

Original

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022 · 2022

Earlier work this paper cites.

Faithful reasoning using large language models

Original

Antonia Creswell and Murray Shanahan. 2022 · 2022

Earlier work this paper cites.

FRUIT: Faithfully reflecting updated information in text

Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022 · 2022

Earlier work this paper cites.

Maieutic prompting: Logically consistent reasoning with recursive explanations

Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022 · 2022

Earlier work this paper cites.

Coderl: Mastering code generation through pretrained models and deep reinforcement learning

Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022 · 2022

Earlier work this paper cites.

Generating training data with language models: Towards zero-shot language understanding

Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022 · 2022

Earlier work this paper cites.

Characteristics of harmful text: Towards rigorous benchmarking of language models

Maribeth Rauh, John F J Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, William Isaac, and Lisa Anne Hendricks. 2022 · 2022

Earlier work this paper cites.

Self-critiquing models for assisting human evaluators

Original

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022 · 2022

Earlier work this paper cites.

Natural language to code translation with execution

Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022 · 2022

Earlier work this paper cites.

Solving math word problems with process- and outcome-based feedback

Original

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022 · 2022

Earlier work this paper cites.

Generating natural language proofs with verifier-guided search

Kaiyu Yang, Jia Deng, and Danqi Chen. 2022a · 2022

Earlier work this paper cites.

Re3: Generating longer stories with recursive reprompting and revision

Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022b · 2022

Earlier work this paper cites.

Star: Bootstrapping reasoning with reasoning

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022 · 2022

Earlier work this paper cites.

RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs

Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. 2023 · 2023

Earlier work this paper cites.

A new era in software security: Towards self-healing software via large language models and formal verification

Original

Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C. Cordeiro. 2023 · 2023

Earlier work this paper cites.

Codet: Code generation with generated tests

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023 · 2023

Earlier work this paper cites.

Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios

Original

I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023 · 2023

Earlier work this paper cites.

Can large language models be an alternative to human evaluations?

Cheng-Han Chiang and Hung-yi Lee. 2023 · 2023

Earlier work this paper cites.

LM vs LM: Detecting factual errors via cross examination

Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023 · 2023

Earlier work this paper cites.

Improving factuality and reasoning in language models through multiagent debate

Original

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023 · 2023

Earlier work this paper cites.

Halo: Estimation and reduction of hallucinations in open-source weak large language models

Original

Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, Shizhu Liu, Pingchuan Tian, Yuping Wang, and Yuxuan Wang. 2023 · 2023

Cited alongside, same era.

From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models

Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023 · 2023

Cited alongside, same era.

Baldur: Whole-proof generation and repair with large language models

Emily First, Markus Rabe, Talia Ringer, and Yuriy Brun. 2023 · 2023

Cited alongside, same era.

RARR: Researching and revising what language models say, using language models

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023 · 2023

Cited alongside, same era.

From wrong to right: A recursive approach towards vision-language explanation

Jiaxin Ge, Sanjay Subramanian, Trevor Darrell, and Boyi Li. 2023 · 2023

Can LLM-generated misinformation be detected?

Canyu Chen and Kai Shu. 2024 · 2024

Closest in time.

Universal self-consistency for large language models

Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2024d · 2024

Closest in time.

Chain-of-verification reduces hallucination in large language models

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024 · 2024

Closest in time.

GPTScore: Evaluate as you desire

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024 · 2024

Closest in time.

CRITIC: Large language models can self-correct with tool-interactive critiquing

Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024 · 2024

Closest in time.

Small language model can self-correct

Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. 2024 · 2024