Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

Understand

Generative AI and large language models hold great promise in enhancing computing education by powering next-generation educational technologies for introductory programming.

• Recent works have studied these models for different scenarios relevant to programming education; however, these works are limited for several reasons, as they typically consider already outdated models or only specific scenario(s).
• Consequently, there is a lack of a systematic study that benchmarks state-of-the-art models for a comprehensive set of programming education scenarios.
• In our work, we systematically evaluate two models, ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human tutors for a variety of scenarios.