Evaluating Language Models for Generating and Judging Programming Feedback

Teaching CS50 with AI: Leveraging Generative Artificial Intelligence in Computer Science Education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2 (Portland, OR, USA) (SIGCSE 2024) . Association for Computing Machinery, New York, NY, USA, 1927

Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J. Malan. 2024b · 1927

Earlier work this paper cites.

Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education . ACM, 41–46

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2016 · 2016

Earlier work this paper cites.

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018 · 2018

Earlier work this paper cites.

Towards understanding the effective design of automated formative feedback for programming assignments

Qiang Hao, David H Smith IV, Lu Ding, Amy Ko, Camille Ottaway, Jack Wilson, Kai H Arakawa, et al · 2022

Earlier work this paper cites.

Automated Assessment in Computer Science Education: A State-of-the-Art Review

José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022 · 2022

Earlier work this paper cites.

Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 . 292–298

Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023 · 2023

Earlier work this paper cites.

Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 . ACM, 93–105

Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023 · 2023

Earlier work this paper cites.

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Original

Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023 · 2023

Earlier work this paper cites.

Mistral 7B

Original

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al · 2023

Earlier work this paper cites.

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proc. of the 2023 CHI Conf. on Human Factors in Computing Systems . ACM, New York, NY, USA, Article 455, 23 pages

Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023 · 2023

Earlier work this paper cites.

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

Original

Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023 · 2023

Earlier work this paper cites.

Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023) . Association for Computing Machinery, New York, NY, USA, 124–130

Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, et al · 2023

Earlier work this paper cites.

G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 2511–2522

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 · 2023

Cited alongside, same era.

Large Language Models (GPT) for automating feedback on programming assignments

Original

Maciej Pankiewicz and Ryan S. Baker. 2023 · 2023

Cited alongside, same era.

Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, et al · 2023

Cited alongside, same era.

Can foundation models label data like humans?

Nazneen Rajani, Nathan Lambert, Sheon Han, Jean Wang, Osvald Nitski, et al · 2023

Cited alongside, same era.

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation

Original

Nachiket Kotalwar, Alkis Gotovos, and Adish Singla. 2024 · 2024

Closest in time.

Using Program Repair as a Proxy for Language Models’ Feedback Ability in Programming Education. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024) , Ekaterina Kochmar, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, and Zheng Yuan (Eds.). Association for Computational Linguistics, Mexico City, Mexico, 165–181

Charles Koutcheme, Nicola Dainese, and Arto Hellas. 2024a · 2024

Closest in time.

Open Source Language Models Can Provide Feedback: Evaluating LLMs’ Ability to Help Students Using GPT-4-As-A-Judge. In Proceedings of the 2024 Innovation and Technology in Computer Science Education, Volume 1 (Milan, Italy) (ITICSE ’24)

Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny. 2024b · 2024

Closest in time.