A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Note on the sampling error of the difference between correlated proportions or percentages

Quinn McNemar. 1947 · 1947

Earlier work this paper cites.

Some statistical problems in measuring the subjective response to drugs

Frederick Mosteller. 1952 · 1952

Earlier work this paper cites.

The framing of decisions and the psychology of choice

Amos Tversky and Daniel Kahneman. 1981 · 1981

Earlier work this paper cites.

Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment

Amos Tversky and Daniel Kahneman. 1983 · 1983

Earlier work this paper cites.

Don’t make your llm an evaluation benchmark cheater

Original

Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023 · 1983

Earlier work this paper cites.

Testing statistical hypotheses , volume 3

Erich Leo Lehmann, Joseph P Romano, and George Casella. 1986 · 1986

Earlier work this paper cites.

Rational choice and the framing of decisions

Amos Tversky and Daniel Kahneman. 1988 · 1988

Earlier work this paper cites.

Controlling the false discovery rate: A practical and powerful approach to multiple testing

Yoav Benjamini and Yosef Hochberg. 1995 · 1995

Earlier work this paper cites.

Rational choice in an uncertain world: The psychology of judgment and decision making

Reid Hastie and Robyn M Dawes. 2009 · 2009

Earlier work this paper cites.

Temporal reasoning on implicit events from distant supervision

Original

Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2020 · 2010

Earlier work this paper cites.

Thinking, fast and slow

Daniel Kahneman. 2011 · 2011

Earlier work this paper cites.

Categorical data analysis , volume 792

Alan Agresti. 2012 · 2012

Earlier work this paper cites.

A corpus and evaluation framework for deeper understanding of commonsense stories

Original

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016 · 2016

Earlier work this paper cites.

Get to the point: Summarization with pointer-generator networks

Original

Abigail See, Peter J Liu, and Christopher D Manning. 2017 · 2017

Earlier work this paper cites.

Commonsenseqa: A question answering challenge targeting commonsense knowledge

Original

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018 · 2018

Earlier work this paper cites.

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Original

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018

Earlier work this paper cites.

Evaluating models’ local decision boundaries via contrast sets

Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020 · 2020

Earlier work this paper cites.

Language models show human-like content effects on reasoning

Original

Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022 · 2022

Earlier work this paper cites.

Towards reasoning in large language models: A survey

Original

Jie Huang and Kevin Chen-Chuan Chang. 2022 · 2022

Earlier work this paper cites.

Logical fallacy detection

Original

Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022 · 2022

Earlier work this paper cites.

Z-icl: zero-shot in-context learning with pseudo-demonstrations

Original

Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022 · 2022

Earlier work this paper cites.

Rethinking the role of demonstrations: What makes in-context learning work?

Original

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Reasoning with language model prompting: A survey

Original

Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022 · 2022

Earlier work this paper cites.

Towards understanding chain-of-thought prompting: An empirical study of what matters

Original

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022 · 2022

Earlier work this paper cites.

Learning to decompose: Hypothetical question decomposition based on comparable texts

Original

Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Using cognitive psychology to understand gpt-3

Marcel Binz and Eric Schulz. 2023 · 2023

Cited alongside, same era.

Sparks of artificial general intelligence: Early experiments with gpt-4

Original

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023 · 2023

Cited alongside, same era.

Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance

Original

Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. 2023 · 2023

Cited alongside, same era.

Rationality of thought improves reasoning in large language models

Tian Gou, Boyao Zhang, Zhenglie Sun, Jing Wang, Yangang Wang, and Jue Wang. 2023 · 2023

Cited alongside, same era.

Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt

Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023 · 2023

Graph of thoughts: Solving elaborate problems with large language models

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024 · 2024

Closest in time.

T2 of thoughts: Temperature tree elicits reasoning in large language models

Original

Chengkun Cai, Xu Zhao, Yucheng Du, Haoliang Liu, and Lei Li. 2024 · 2024

Closest in time.

Premise order matters in reasoning with large language models

Original

Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024 · 2024

Closest in time.

Cognitive bias in high-stakes decision-making with llms

Original

Jessica Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024 · 2024

Closest in time.

Puzzle solving using reasoning of large language models: A survey