Fetching the paper…

Reading the bibliography…

2023

GPT-4 Can't Reason

Arkoudas, Konstantine

Original
PDF
Post on X

Understand

GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI's previously best model, which had powered the initial release of ChatGPT).

However, despite the genuinely impressive improvement, there are good reasons to be highly skeptical of GPT-4's ability to reason.
This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community, as well as the way in which LLM reasoning performance is currently evaluated; introduces a small collection of 21 diverse reasoning problems; and performs a detailed qualitative evaluation of GPT-4's performance on those problems.
Based on this analysis, the paper concludes that, despite its occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.

GPT-4 Can't Reason · Around

Built on

Situations and Attitudes
Barwise, J. and Perry, J., · 1983
Earlier work this paper cites.
How We Reason
Johnson-Laird, P.N., · 2006
Earlier work this paper cites.
Temporal Reasoning Problems and Algorithms for Solving Them: Literature Survey
Planken, L., · 2008
Earlier work this paper cites.
Fundamental Proof Methods in Computer Science
Arkoudas, K. and Musser, D., · 2017
Earlier work this paper cites.

Similar

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, T., Pavlick, E., Linzen, T., · 2019
Cited alongside, same era.
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., Kao, H.-Y · 2019
Cited alongside, same era.
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., … \ldots , Tenenholtz, M., · 2022
Cited alongside, same era.
ReAct: Synergizing Reasoning and Acting in Language Models
Original
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y., · 2023
Cited alongside, same era.

Then

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
Liu H., Ning R., Teng, Z., Liu, J., Zhou, Q., Zhang, Y., · 2023
Closest in time.
GPT-4 Technical Report
OpenAI, · 2023
Closest in time.
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
Wang, J., Hu, X., Hou, W., Chen, H., Zheng, R., Wang, Y., … \ldots , Xie, X., · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…