Fetching the paper…
Reading the bibliography…
This paper studies recent developments in large language models' (LLM) abilities to pass assessments in introductory and intermediate Python programming courses at the postsecondary level.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901
Earlier work this paper cites.
Literate programming
Donald Ervin Knuth. 1984 · 1984
Earlier work this paper cites.
Measuring Massive Multitask Language Understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009
Earlier work this paper cites.
Making Pre-trained Language Models Better Few-shot Learners
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021 · 2012
Earlier work this paper cites.
Project-based learning: A review of the literature
Dimitra Kokotsaki, Victoria Menzies, and Andy Wiggins. 2016 · 2016
Earlier work this paper cites.
A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 839–849
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016 · 2016
Earlier work this paper cites.
Race: Large-scale reading comprehension dataset from examinations
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Can a suit of armor conduct electricity? A new dataset for open book question answering
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Earlier work this paper cites.
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021
Earlier work this paper cites.
Solving Linear Algebra by Program Synthesis
Iddo Drori and Nakul Verma. 2021 · 2021
Earlier work this paper cites.
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Laria Reynolds and Kyle McDonell. 2021 · 2021
Earlier work this paper cites.
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021 · 2021
Earlier work this paper cites.
Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2022 · 2022
Earlier work this paper cites.
Michael Bommarito II and Daniel Martin Katz. 2022 · 2022
Earlier work this paper cites.
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22) . Association for Computing Machinery, New York, NY, USA, 10–19
James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022 · 2022
Cited alongside, same era.
How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv
Amelia Gilson, Conrad W. Safranek, Tao Huang, Vimig Socrates, Lim Sze Chi, Roderick A. Taylor, and David Chartash. 2022 · 2022
Cited alongside, same era.
Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation
Anjan Karmakar, Julian Aron Prenner, Marco D’Ambros, and Romain Robbes. 2022 · 2022
Cited alongside, same era.
Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models
Tiffany H Kung, Morgan Cheatham, Arielle Medinilla, Czarina Sillos, Lorie De Leon, Camille Elepano, Marie Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al · 2022
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023 · 2023
Closest in time.
Conversing with Copilot: Exploring prompt engineering for solving CS1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 1136–1142
Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023 · 2023
Closest in time.
Giuseppe Destefanis, Silvia Bartolucci, and Marco Ortu. 2023 · 2023
Closest in time.
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference . 97–104
James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023 · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Competition-level code generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022 · 2022
Cited alongside, same era.
Can large language models reason about medical questions?
Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022 · 2022
Cited alongside, same era.
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 · 2022
Cited alongside, same era.
Explainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work?
Ahmad Haji Mohammadkhani, Chakkrit Kla Tantithamthavorn, and Hadi Hemmati. 2022 · 2022
Cited alongside, same era.
An Empirical Evaluation of GitHub Copilot’s Code Suggestions. In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) . 1–5
Nhan Nguyen and Sarah Nadi. 2022 · 2022
Cited alongside, same era.
Training language models to follow instructions with human feedback
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Cited alongside, same era.
Do Users Write More Insecure Code with AI Assistants?
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2022 · 2022
Cited alongside, same era.
Leveraging Large Language Models for Multiple Choice Question Answering
Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022 · 2022
Cited alongside, same era.
Closest in time.
ChatGPT and Software Testing Education: Promises & Perils
Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023 · 2023
Closest in time.
Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023 · 2023
Closest in time.
GPT-4 Passes the Bar Exam
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023 · 2023
Closest in time.
Studying the Effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23) . Association for Computing Machinery, New York, NY, USA, Article 455, 23 pages
Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023 · 2023
Closest in time.
Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023) . Association for Computing Machinery, New York, NY, USA, 563–569
Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023b · 2023
Closest in time.
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023) . Association for Computing Machinery, New York, NY, USA, 931–937
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023 · 2023
Closest in time.
Augmented Language Models: a Survey
Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al · 2023
Closest in time.
OpenAI. 2023 · 2023
Closest in time.
Many bioinformatics programming tasks can be automated with ChatGPT
Stephen R Piccolo, Paul Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G Ridge. 2023 · 2023
Closest in time.
James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023 · 2023
Closest in time.
Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr. 2023a · 2023
Closest in time.
Prompting Large Language Model for Machine Translation: A Case Study
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023 · 2023
Closest in time.