Built on

Collins, Harry M. “Embedded or embodied? A review of Hubert Dreyfus’ what computers still can’t do.” Artificial Intelligence 80.1: 99-118
1996
Earlier work this paper cites.
Asher, Nicholas, et al. “Limits for Learning with Language Models,” arXiv preprint arXiv:2306.12213
Original
2023
Earlier work this paper cites.
K. Zhou, Y. Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y. Lin, J.-R. Wen, and J. Han, “Don’t Make Your LLM an Evaluation Benchmark Cheater,” arXiv preprint arXiv:2311.01964
Original
2023
Earlier work this paper cites.
S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation,” arXiv preprint arXiv:2308.02828
Original
2023
Earlier work this paper cites.
Davis, E., et al. “Mathematics, word problems, common sense, and artificial intelligence,” Bulletin of the American Mathematical Society
2024
Earlier work this paper cites.
Miyu Sasaki, Natsumi Watanabe, Tsukihito Komanaka et al. “Enhancing Contextual Understanding of Mistral LLM with External Knowledge Bases,” Research Square rs.3.rs-4215447/v1
2024
Earlier work this paper cites.

Similar

Wu, Wenshan, et al. “Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models,” arXiv preprint arXiv:2404.03622
Original
2024
Cited alongside, same era.
Ahn, Janice, et al. “Large Language Models for Mathematical Reasoning: Progresses and Challenges,” arXiv preprint arXiv:2402.00157
Original
2024
Cited alongside, same era.
Wachter, S., Mittelstadt, B., et al. “Do large language models have a legal duty to tell the truth?,” Available at SSRN 4771884
2024
Cited alongside, same era.
Li, Zhiming, et al. “LLMs for Relational Reasoning: How Far are We?” arXiv preprint arXiv:2401.09042
Original
2024
Cited alongside, same era.
Giadikiaroglou, Panagiotis, et al.“Puzzle Solving using Reasoning of Large Language Models: A Survey,” arXiv preprint arXiv:2402.11291

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Easy Problems That LLMs Get Wrong

Williams, Sean, Huckle, James

Understand

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others.

Through a series of straightforward questions, it uncovers the significant limitations of well-regarded models to perform tasks that humans manage with ease.
It also highlights the potential of prompt engineering to mitigate some errors and underscores the necessity for better training methodologies.
Our findings stress the importance of grounding LLMs with human reasoning and common sense, emphasising the need for human-in-the-loop for enterprise applications.

Built on

Collins, Harry M. “Embedded or embodied? A review of Hubert Dreyfus’ what computers still can’t do.” Artificial Intelligence 80.1: 99-118
1996
Earlier work this paper cites.
Asher, Nicholas, et al. “Limits for Learning with Language Models,” arXiv preprint arXiv:2306.12213
Original
2023
Earlier work this paper cites.
K. Zhou, Y. Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y. Lin, J.-R. Wen, and J. Han, “Don’t Make Your LLM an Evaluation Benchmark Cheater,” arXiv preprint arXiv:2311.01964
Original
2023
Earlier work this paper cites.
S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation,” arXiv preprint arXiv:2308.02828
Original
2023
Earlier work this paper cites.
Davis, E., et al. “Mathematics, word problems, common sense, and artificial intelligence,” Bulletin of the American Mathematical Society
2024
Earlier work this paper cites.
Miyu Sasaki, Natsumi Watanabe, Tsukihito Komanaka et al. “Enhancing Contextual Understanding of Mistral LLM with External Knowledge Bases,” Research Square rs.3.rs-4215447/v1
2024
Earlier work this paper cites.

Similar

Wu, Wenshan, et al. “Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models,” arXiv preprint arXiv:2404.03622
Original
2024
Cited alongside, same era.
Ahn, Janice, et al. “Large Language Models for Mathematical Reasoning: Progresses and Challenges,” arXiv preprint arXiv:2402.00157
Original
2024
Cited alongside, same era.
Wachter, S., Mittelstadt, B., et al. “Do large language models have a legal duty to tell the truth?,” Available at SSRN 4771884
2024
Cited alongside, same era.
Li, Zhiming, et al. “LLMs for Relational Reasoning: How Far are We?” arXiv preprint arXiv:2401.09042
Original
2024
Cited alongside, same era.
Giadikiaroglou, Panagiotis, et al.“Puzzle Solving using Reasoning of Large Language Models: A Survey,” arXiv preprint arXiv:2402.11291

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2024

Cited alongside, same era.

Original

2024

Closest in time.