Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 1901

Earlier work this paper cites.

Reasoning about a rule

Peter C. Wason. 1968 · 1968

Earlier work this paper cites.

Psychology of reasoning: Structure and content , volume 86

Peter Cathcart Wason and Philip Nicholas Johnson-Laird. 1972 · 1972

Earlier work this paper cites.

Reasoning about Knowledge

Ronald Fagin, Joseph Y. Halpern, Yoram Moses, and Moshe Vardi. 2004 · 2004

Earlier work this paper cites.

Chapter 5 - cryptography

Jason Andress. 2014 · 2014

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Does learning require memorization? A short tale about a long tail

Vitaly Feldman. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

Data distributional properties drive emergent in-context learning in transformers

Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022 · 2022

Earlier work this paper cites.

Folio: Natural language reasoning with first-order logic

Original

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

Text and patterns: For effective chain of thought, it takes two to tango

Original

Aman Madaan and Amir Yazdanbakhsh. 2022 · 2022

Earlier work this paper cites.

Data contamination: From memorization to exploitation

Inbal Magar and Roy Schwartz. 2022 · 2022

Earlier work this paper cites.

Rethinking the role of demonstrations: What makes in-context learning work?

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Impact of pretraining term frequencies on few-shot numerical reasoning

Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2022 · 2022

Earlier work this paper cites.

Chain of thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

An explanation of in-context learning as implicit Bayesian inference

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022 · 2022

Cited alongside, same era.

The unreliability of explanations in few-shot prompting for textual reasoning

Xi Ye and Greg Durrett. 2022 · 2022

Cited alongside, same era.

Towards revealing the mystery behind chain of thought: A theoretical perspective

Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. 2023 · 2023

Cited alongside, same era.

In-context learning creates task vectors

Roee Hendel, Mor Geva, and Amir Globerson. 2023 · 2023

Cited alongside, same era.

Towards reasoning in large language models: A survey

Jie Huang and Kevin Chen-Chuan Chang. 2023 · 2023

Cited alongside, same era.

Can large language models truly understand prompts? A case study with negated prompts

Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2023 · 2023

Complementary explanations for effective in-context learning

Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Veselin Stoyanov, Greg Durrett, and Ramakanth Pasunuru. 2023 · 2023

Later among the works it cites.

Large language models are human-level prompt engineers

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023 · 2023

Later among the works it cites.

Claude 3

Anthropic. 2024 · 2024

Closest in time.

Generalization vs. memorization: Tracing language models’ capabilities back to pretraining data

Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2024 · 2024

Closest in time.

Retentive or forgetful? Diving into the knowledge memorizing mechanism of language models

Boxi Cao, Qiaoyu Tang, Hongyu Lin, Shanshan Jiang, Bin Dong, Xianpei Han, Jiawei Chen, Tianshu Wang, and Le Sun. 2024 · 2024

Closest in time.

Understanding in-context learning with a pelican soup framework