2021

Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs

Prenner, Julian Aron, Robbes, Romain

Understand

OpenAI's Codex, a GPT-3 like model trained on a large code corpus, has made headlines in and outside of academia.

Given a short user-provided description, it is capable of synthesizing code snippets that are syntactically and semantically valid in most cases.
In this work, we want to investigate whether Codex is able to localize and fix bugs, a task of central interest in the field of automated program repair.
Our initial evaluation uses the multi-language QuixBugs benchmark (40 bugs in both Python and Java).

“Language Models Are Few-Shot Learners”, 2020
Original
Tom. Brown et al · 2005
Earlier work this paper cites.
“Maintaining Mental Models: A Study of Developer Work Habits”
Thomas. LaToza, Gina Venolia and Robert DeLine · 2006
Earlier work this paper cites.
“GenProg: A Generic Method for Automatic Software Repair”
C. Goues et al · 2011
Earlier work this paper cites.
“Automatic Patch Generation Learned from Human-Written Patches”
Dongsun Kim et al · 2013
Earlier work this paper cites.
“Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs”
René Just, Darioush Jalali and Michael. Ernst · 2014
Earlier work this paper cites.
“Elixir: Effective Object-Oriented Program Repair”
R.. Saha et al · 2017
Earlier work this paper cites.
“Leveraging Syntax-Related Code for Automated Program Repair”
Qi Xin and Steven. Reiss · 2017
Earlier work this paper cites.
“QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge”
Derrick Lin et al · 2017
Earlier work this paper cites.

“Attention Is All You Need”, 2017
Original
Ashish Vaswani et al · 2017
Cited alongside, same era.
“ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming”
Yuan Yuan and Wolfgang Banzhaf · 2018
Cited alongside, same era.
“LSRepair: Live Search of Fix Ingredients for Automated Program Repair”
Kui Liu et al · 2018
Cited alongside, same era.
“TBar: Revisiting Template-Based Automated Program Repair”
Kui Liu et al · 2019
Cited alongside, same era.
“SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair”, 2019
Zimin Chen et al · 2019
Cited alongside, same era.
“BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”, 2019
Original
Jacob Devlin et al · 2019
Cited alongside, same era.
“BugsJS: A Benchmark of JavaScript Bugs”
Péter Gyimesi et al · 2019
Cited alongside, same era.
“Reversible Debugging Software “Quantify the Time and Cost Saved Using Reversible Debuggers””
Tom Britton et al
Cited in the paper.

“HOPPITY: LEARNING GRAPH TRANSFORMATIONS TO DETECT AND FIX BUGS IN PROGRAMS”, 2020
Elizabeth Dinella et al · 2020
Later among the works it cites.
“CoCoNuT: Combining Context-Aware Neural Translation Models Using Ensemble for Program Repair”
Thibaud Lutellier et al · 2020
Later among the works it cites.
“Scaling laws for neural language models”
Original
Jared Kaplan et al · 2020
Later among the works it cites.
“CURE: Code-Aware Neural Machine Translation for Automatic Program Repair”, 2021
Original
Nan Jiang, Thibaud Lutellier and Lin Tan · 2021
Closest in time.
“Evaluating Large Language Models Trained on Code”, 2021
Original
Mark Chen et al · 2021
Closest in time.
“DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons”, 2021
Original
Dawn Drain et al · 2021
Closest in time.
“Research recitation: A first look at rote learning in GitHub Copilot suggestions”, 2021
Albert Ziegler · 2021
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…