Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics

Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds.). Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009

Earlier work this paper cites.

Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling ’10) . Association for Computing Machinery, New York, NY, USA, 86–93

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010 · 2010

Earlier work this paper cites.

Where Has The Time Gone? Faculty Activities and Time Commitments in the Online Classroom

B. Mandernach, Swinton Hudson, and Shanna Wise. 2013 · 2013

Earlier work this paper cites.

Automated Assessment of Programming Assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research (Arnhem, Netherlands) (CSERC ’13) . Open Universiteit, Heerlen, Heerlen, NLD, 45–56

Vreda Pieterse. 2013 · 2013

Earlier work this paper cites.

Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE ’16) . Association for Computing Machinery, New York, NY, USA, 41–46

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2016 · 2016

Earlier work this paper cites.

Coderunner: a tool for assessing computer programming skills

Richard Lobb and Jenny Harlow. 2016 · 2016

Earlier work this paper cites.

On Novices’ Interaction with Compiler Error Messages: A Human Factors Approach. In Proceedings of the 2017 ACM Conference on International Computing Education Research (Tacoma, Washington, USA) (ICER ’17) . Association for Computing Machinery, New York, NY, USA, 74–82

James Prather, Raymond Pettit, Kayla Holcomb McMurry, Alani Peters, John Homer, Nevan Simone, and Maxine Cohen. 2017 · 2017

Earlier work this paper cites.

Application of Rubrics in the Classroom: A Vital Tool for Improvement in Assessment, Feedback and Learning

Faieza Chowdhury. 2018 · 2018

Earlier work this paper cites.

A Systematic Literature Review of Automated Feedback Generation for Programming Exercises

Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

How Automated Feedback is Delivered Matters: Formative Feedback and Knowledge Transfer. In 2019 IEEE Frontiers in Education Conference (FIE) . 1–6

Qiang Hao and Michail Tsikerdekis. 2019 · 2019

Earlier work this paper cites.

Automatic Grading of Programming Assignments: An Approach Based on Formal Semantics. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) . 126–137

Xiao Liu, Shuai Wang, Pei Wang, and Dinghao Wu. 2019 · 2019

Earlier work this paper cites.

Evaluation Rubric for Computational Thinking Concepts. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT) , Vol. 2161-377X. 279–281

Christiano Otero Avila, Luciana Foss, Adriana Bordini, Maria Simone Debacco, and Simone André da Costa Cavalheiro. 2019 · 2019

Earlier work this paper cites.

Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 (ICER 2022) . ACM, 27–43

Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022 · 2022

Earlier work this paper cites.

Expectation vs Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22) . Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages

Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022 · 2022

Cited alongside, same era.

Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (ICER 2023) . ACM, 93–105

Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023 · 2023

Cited alongside, same era.

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Original

Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023 · 2023

Cited alongside, same era.

"With Great Power Comes Great Responsibility!": Student and Instructor Perspectives on the influence of LLMs on Undergraduate Engineering Education

Feedback-Generation for Programming Exercises With GPT-4. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2024) . ACM, 31–37

Imen Azaiz, Natalie Kiesler, and Sven Strickroth. 2024 · 2024

Later among the works it cites.

Experiences from Integrating Large Language Model Chatbots into the Classroom

Original

Arto Hellas, Juho Leinonen, and Leo Leppänen. 2024 · 2024

Later among the works it cites.

Evaluating the Application of Large Language Models to Generate Feedback in Programming Education. In 2024 IEEE Global Engineering Education Conference (EDUCON) . IEEE, 1–5

Sven Jacobs and Steffen Jaschke. 2024 · 2024

Later among the works it cites.

ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Portland, OR, USA) (SIGCSE 2024) . Association for Computing Machinery, New York, NY, USA, 625–631

Ishika Joshi, Ritvik Budhiraja, Harshal Dev, Jahnvi Kadia, Mohammad Osama Ataullah, Sayan Mitra, Harshal D. Akolekar, and Dhruv Kumar. 2024 · 2024