Fetching the paper…

Reading the bibliography…

2023

An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)

Shakarian, Paulo, Koyyalamudi, Abhinav, Ngu, Noel et al.

Original
PDF
Post on X

Understand

We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K.

To our knowledge, this is the first independent evaluation of ChatGPT.
We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not.
Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations.

An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP) · Around

Built on

Learning from explicit and implicit supervision jointly for algebra word problems,
S. Upadhyay, M.-W. Chang, K.-W. Chang, W.-t. Yih, · 2016
Earlier work this paper cites.
S. Upadhyay, M.-W. Chang, Annotating derivations: A new evaluation strategy and dataset for algebra word problems, URL: http://arxiv.org/abs/1609.07197 . doi: 10.48550/arXiv.1609.07197
Original
2017
Earlier work this paper cites.
Failing to learn: Autonomously identifying perception failures for self-driving cars 3 3860–3867. URL: http://arxiv.org/abs/1707.00051 . doi: 10.1109/LRA.2018.2857402 . arXiv:1707.00051 [cs]
Original
M. S. Ramanagopal, C. Anderson, R. Vasudevan, M. Johnson-Roberson, · 2018
Earlier work this paper cites.
Generating equation by utilizing operators : GEO model,
K. S. Ki, D. Lee, B. Kim, G. Gweon, · 2020
Earlier work this paper cites.
Point to the expression: Solving algebraic word problems using the expression-pointer transformer model,
Original
B. Kim, K. S. Ki, D. Lee, G. Gweon, · 2020
Earlier work this paper cites.

Similar

Meta-learning in neural networks: A survey 44 5149–5169. URL: https://www.computer.org/csdl/journal/tp/2022/09/09428530/1twaJR3AcJW . doi: 10.1109/TPAMI.2021.3079209 , publisher: IEEE Computer Society
T. Hospedales, A. Antoniou, P. Micaelli, A. Storkey, · 2021
Cited alongside, same era.
How does gpt obtain its ability? tracing emergent abilities of language models to their sources, URL: https://yaofu.notion.site
2022
Cited alongside, same era.
Chatgpt: Optimizing language models for dialogue, URL: https://openai.com/blog/chatgpt/
2022
Cited alongside, same era.
Chatgpt gained 1 million users in under a week. here’s why the AI chatbot is primed to disrupt search as we know it. URL: https://www.yahoo.com/video/chatgpt-gained-1-million-followers-224523258.html
2022
Cited alongside, same era.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre, Training compute-optimal large language models, URL: http://arxiv.org/abs/2203.15556 . arXiv:2203.15556 [cs]

Then

Domain generalization: A survey 1–20. doi: 10.1109/TPAMI.2022.3195549 , conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence
K. Zhou, Z. Liu, Y. Qiao, T. Xiang, C. C. Loy, · 2022
Later among the works it cites.
Continual pre-training of language models for math problem understanding with syntax-aware memory network,
Z. Gong, K. Zhou, X. Zhao, J. Sha, S. Wang, J.-R. Wen, · 2022
Later among the works it cites.
EPT-x: An expression-pointer transformer model that generates eXplanations for numbers,
B. Kim, K. S. Ki, S. Rhim, G. Gweon, · 2022
Later among the works it cites.
ReasonFuse: Reason path driven and global–local fusion network for numerical table-text question answering 516 169–181. URL: https://www.sciencedirect.com/science/article/pii/S0925231222011444 . doi: 10.1016/j.neucom.2022.09.046
Y. Xia, F. Li, Q. Liu, L. Jin, Z. Zhang, X. Sun, L. Shao, · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited in the paper.

Chain-of-thought prompting elicits reasoning in large language models

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou,

Cited in the paper.

X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency improves chain of thought reasoning in language models, URL: http://arxiv.org/abs/2203.11171 . doi: 10.48550/arXiv.2203.11171 . arXiv:2203.11171 [cs]

Cited in the paper.

S. Daftry, S. Zeng, J. A. Bagnell, M. Hebert, Introspective perception: Learning to predict failures in vision systems, URL: http://arxiv.org/abs/1607.08665 . doi: 10.48550/arXiv.1607.08665 . arXiv:1607.08665 [cs]

Cited in the paper.

MWPToolkit: An open-source framework for deep learning-based math word problem solvers 36 13188–13190. URL: https://ojs.aaai.org/index.php/AAAI/article/view/21723 . doi: 10.1609/aaai.v36i11.21723 , number: 11

Y. Lan, L. Wang, Q. Zhang, Y. Lan, B. T. Dai, Y. Wang, D. Zhang, E.-P. Lim,

Cited in the paper.