2023

Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code

Savelka, Jaromir, Agarwal, Arav, Bogart, Christopher et al.

Understand

We analyzed effectiveness of three generative pre-trained transformer (GPT) models in answering multiple-choice question (MCQ) assessments, often involving short snippets of code, from introductory and intermediate programming courses at the postsecondary level.

This emerging technology stirs countless discussions of its potential uses (e.g., exercise generation, code explanation) as well as misuses in programming education (e.g., cheating).
However, the capabilities of GPT models and their limitations to reason about and/or analyze code in educational settings have been under-explored.
We evaluated several OpenAI's GPT models on formative and summative MCQ assessments from three Python courses (530 questions).

Built on

Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · 1901
Earlier work this paper cites.
Literate programming
Knuth, D. E. (1984) · 1984
Earlier work this paper cites.
Extreme programming explained: embrace change
Beck, K. (2000) · 2000
Earlier work this paper cites.
Software design–cognitive aspects
Détienne, F. and Bott, F. (2002) · 2002
Earlier work this paper cites.
The effects of pair-programming on performance in an introductory programming course
McDowell, C., Werner, L., Bullock, H., and Fernald, J. (2002) · 2002
Earlier work this paper cites.
Learning and Teaching Programming: A Review and Discussion
Robins, A., Rountree, J., and Rountree, N. (2003) · 2003
Earlier work this paper cites.
Measuring massive multitask language understanding
Original
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020) · 2009
Earlier work this paper cites.
A corpus and cloze evaluation for deeper understanding of commonsense stories
Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., and Allen, J. (2016) · 2016
Earlier work this paper cites.
Race: Large-scale reading comprehension dataset from examinations
Original
Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. (2017) · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017) · 2017
Earlier work this paper cites.
Can a suit of armor conduct electricity? A new dataset for open book question answering
Original
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018) · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018) · 2018
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019) · 2019
Earlier work this paper cites.

Similar

Evaluating large language models trained on code
Original
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021) · 2021
Cited alongside, same era.
Solving linear algebra by program synthesis
Original
Drori, I. and Verma, N. (2021) · 2021
Cited alongside, same era.
Introducing Amazon CodeWhisperer, the ML-powered coding companion
Ankur, D. and Atul, D. (2022) · 2022
Cited alongside, same era.
Programming is hard–or at least it used to be: Educational opportunities and challenges of ai code generation
Original
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., and Santos, E. A. (2022) · 2022
Cited alongside, same era.
Fooling moss detection with pretrained language models
Biderman, S. R. and Raff, E. (2022) · 2022
Cited alongside, same era.
GPT takes the bar exam
Original
Bommarito II, M. and Katz, D. M. (2022) · 2022
Cited alongside, same era.
Conversing with Copilot: Exploring prompt engineering for solving cs1 problems using natural language
Original
Denny, P., Kumar, V., and Giacaman, N. (2022) · 2022
Cited alongside, same era.
The robots are coming: Exploring the implications of OpenAI Codex on introductory programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., and Prather, J. (2022) · 2022
Cited alongside, same era.
How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L. S., Taylor, R. A., and Chartash, D. (2022) · 2022
Cited alongside, same era.
Codex hacks HackerRank: Memorization issues and a framework for code synthesis evaluation
Original
Karmakar, A., Prenner, J. A., D’Ambros, M., and Robbes, R. (2022) · 2022
Cited alongside, same era.
Performance of ChatGPT on USMLE: Potential for ai-assisted medical education using large language models
Kung, T. H., Cheatham, M., Medinilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2022) · 2022
Cited alongside, same era.
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O. (2022) · 2022
Cited alongside, same era.

Then

Learn to explain: Multimodal reasoning via thought chains for science question answering
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. (2022) · 2022
Later among the works it cites.
Explainable AI for pre-trained code models: What do they learn? When they do not work?
Original
Mohammadkhani, A. H., Tantithamthavorn, C. K., and Hemmati, H. (2022) · 2022
Later among the works it cites.
An empirical evaluation of GitHub Copilot’s code suggestions
Nguyen, N. and Nadi, S. (2022) · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Original
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · 2022
Later among the works it cites.
Leveraging large language models for multiple choice question answering
Original
Robinson, J., Rytting, C. M., and Wingate, D. (2022) · 2022
Later among the works it cites.
How important are good method names in neural code generation? A model robustness perspective
Original
Yang, G., Zhou, Y., Yang, W., Yue, T., Chen, X., and Chen, T. (2022) · 2022
Later among the works it cites.
GPT as knowledge worker: A zero-shot evaluation of (AI) CPA capabilities
Original
Bommarito, J., Bommarito, M., Katz, D. M., and Katz, J. (2023) · 2023
Closest in time.
A college student created an app that can tell whether ai wrote an essay
Bowman, E. (2023) · 2023
Closest in time.
NYC education department blocks ChatGPT on school devices, networks
Elsen-Rooney, M. (2023) · 2023
Closest in time.
Alarmed by A.I. chatbots, universities start revamping how they teach
Huang, K. (2023) · 2023
Closest in time.
Augmented language models: a survey
Original
Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. (2023) · 2023
Closest in time.
Can generative pre-trained transformers (gpt) pass assessments in higher education programming courses?
Savelka, J., Agarwal, A., Bogart, C., Song, Y., and Sakr, M. (2023) · 2023
Closest in time.
Using GitHub Copilot to solve simple programming problems
Wermelinger, M. (2023) · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…