How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark

Effects of noise letters upon the identification of a target letter in a nonsearch task

Barbara A. Eriksen and Charles W. Eriksen. 1974 · 1974

Earlier work this paper cites.

Proofwriter: Generating implications, proofs, and abductive statements over natural language

Original

Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2021 · 2012

Earlier work this paper cites.

ProtoQA: A question answering dataset for prototypical common-sense reasoning

Michael Boratko, Xiang Li, Tim O’Gorman, Rajarshi Das, Dan Le, and Andrew McCallum. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Solving quantitative reasoning problems with language models

Original

Aitor Lewkowycz, Eric Zelikman, Ran Tao, Tatsunori Hashimoto, John Schulman, Xinyun Chen, C Lawrence Zitnick, Christopher D Manning, Daniel Saunders, Adam Santoro, et al. 2022 · 2022

Earlier work this paper cites.

Solving math word problems with process- and outcome-based feedback

Original

Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022 · 2022

Earlier work this paper cites.

Star: Bootstrapping reasoning with reasoning

Original

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022 · 2022

Earlier work this paper cites.

Least-to-most prompting enables complex reasoning in large language models

Original

Xuezhi Zhou, Nathanael Schärli, Luheng Hou, Jason Wei, Swaroop Mishra, Xinyun Huang, Quoc Le, Hyung Won Chung, Hieu Pham, Barret Zoph, et al. 2022 · 2022

Earlier work this paper cites.

Art: Automatic multi-step reasoning and tool-use for large language models

Original

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023 · 2023

Earlier work this paper cites.

Toolformer: Language models can teach themselves to use tools

Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023 · 2023

Earlier work this paper cites.

Large language models can be easily distracted by irrelevant context

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023a · 2023

Earlier work this paper cites.

Language models can improve event prediction by few-shot abductive reasoning

Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Y. Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. 2023b · 2023

Earlier work this paper cites.

Self-consistency improves chain of thought reasoning in language models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 · 2023

Earlier work this paper cites.

Qa-lora: Quantization-aware low-rank adaptation of large language models

Original

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. 2023 · 2023

Earlier work this paper cites.

Tree of thoughts: Deliberate problem solving with large language models

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023b · 2023

Cited alongside, same era.

React: Synergizing reasoning and acting in language models

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023c · 2023

Cited alongside, same era.

Cutting through the noise: Boosting llm performance on math word problems

Original

Ujjwala Anantheswaran, Himanshu Gupta, Kevin Scaria, Shreyas Verma, Chitta Baral, and Swaroop Mishra. 2024 · 2024

Cited alongside, same era.

The reversal curse: Llms trained on "a is b" fail to learn "b is a"

Original

Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2024 · 2024

Cited alongside, same era.

A systematic analysis of large language models as soft reasoners: The case of syllogistic inferences

Rag-instruct: Boosting llms with diverse retrieval-augmented instructions

Original

Wanlong Liu, Junying Chen, Ke Ji, Li Zhou, Wenyu Chen, and Benyou Wang. 2024 · 2024

Later among the works it cites.

How much do prompting methods help llms on quantitative reasoning with irrelevant information?

Seok Hwan Song and Wallapak Tavanapong. 2024 · 2024

Later among the works it cites.

How easily do irrelevant inputs skew the responses of large language models?

Original

Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024 · 2024

Later among the works it cites.

Pride and prejudice: Llm amplifies self-bias in self-refinement

Original

Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. 2024 · 2024

Later among the works it cites.

Processbench: Identifying process errors in mathematical reasoning