2022

FixEval: Execution-based Evaluation of Program Fixes for Programming Problems

Haque, Md Mahim Anjum, Ahmad, Wasi Uddin, Lourentzou, Ismini et al.

Understand

The complexity of modern software has led to a drastic increase in the time and cost associated with detecting and rectifying software bugs.

In response, researchers have explored various methods to automatically generate fixes for buggy code.
However, due to the large combinatorial space of possible fixes for any given bug, few tools and datasets are available to evaluate model-generated fixes effectively.
To address this issue, we introduce FixEval, a benchmark comprising of buggy code submissions to competitive programming problems and their corresponding fixes.

Built on

The economic impacts of inadequate infrastructure for software testing
National Institute of Standards and Technology · 2002
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu · 2002
Earlier work this paper cites.
https://judge.u-aizu.ac.jp/onlinejudge
Aizu Online Judge, 2004 · 2004
Earlier work this paper cites.
On the automation of fixing software bugs
A. Arcuri · 2008
Earlier work this paper cites.
Evolving patches for software repair
T. Ackling, B. Alexander, and I. Grunert · 2011
Earlier work this paper cites.
A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each
C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer · 2012
Earlier work this paper cites.
On the complexity of software systems
T. Mens · 2012
Earlier work this paper cites.
Reversible debugging software-quantify the time and cost saved using reversible debuggers
T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen · 2013
Earlier work this paper cites.
Automatic patch generation learned from human-written patches
D. Kim, J. Nam, J. Song, and S. Kim · 2013
Earlier work this paper cites.
Automatic repair of buggy if conditions and missing preconditions with smt
F. DeMarco, J. Xuan, D. Le Berre, and M. Monperrus · 2014
Earlier work this paper cites.
Defects4j: A database of existing faults to enable controlled testing studies for java programs
R. Just, D. Jalali, and M. D. Ernst · 2014
Earlier work this paper cites.
Programmers’ build errors: A case study (at google)
H. Seo, C. Sadowski, S. Elbaum, E. Aftandilian, and R. Bowdidge · 2014
Earlier work this paper cites.
The ManyBugs and IntroClass benchmarks for automated repair of C programs
C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P. Devanbu, S. Forrest, and W. Weimer · 2015
Earlier work this paper cites.
Deepfix: Fixing common c language errors by deep learning
R. Gupta, S. Pal, A. Kanade, and S. Shevade · 2017
Earlier work this paper cites.

Similar

Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge
D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama · 2017
Cited alongside, same era.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017
Cited alongside, same era.
Automatic software repair: A bibliography
M. Monperrus · 2018
Cited alongside, same era.
Automated program repair
C. L. Goues, M. Pradel, and A. Roychoudhury · 2019
Cited alongside, same era.
Re-factoring based program repair applied to programming assignments
Y. Hu, U. Z. Ahmed, S. Mechtaev, B. Leong, and A. Roychoudhury · 2019
Cited alongside, same era.
Spoc: Search-based pseudocode to code
S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang · 2019
Cited alongside, same era.
Decoupled weight decay regularization
I. Loshchilov and F. Hutter · 2019
Cited alongside, same era.
Deepdelta: Learning to repair compilation errors
A. Mesbah, A. Rice, E. Johnston, N. Glorioso, and E. Aftandilian · 2019
Cited alongside, same era.
Megatron-lm: Training multi-billion parameter language models using model parallelism
Original
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro · 2019
Cited alongside, same era.
On learning meaningful code changes via neural machine translation
M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk · 2019
Cited alongside, same era.
Atcoder opensourced test cases
Atcoder · 2020
Cited alongside, same era.
Language models are few-shot learners
T. Brown, B. Mann, N. Ryder, M. Subbiah, et al · 2020
Cited alongside, same era.

Then

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2020
Later among the works it cites.
Codebleu: a method for automatic evaluation of code synthesis
Original
S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma · 2020
Later among the works it cites.
Unified pre-training for program understanding and generation
W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang · 2021
Later among the works it cites.
Avatar: A parallel corpus for java-python program translation
Original
W. U. Ahmad, M. G. R. Tushar, S. Chakraborty, and K.-W. Chang · 2021
Later among the works it cites.
Tfix: Learning to fix coding errors with a text-to-text transformer
B. Berabi, J. He, V. Raychev, and M. Vechev · 2021
Later among the works it cites.
Evaluating large language models trained on code
Original
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al · 2021
Later among the works it cites.
Measuring coding challenge competence with APPS
D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt · 2021
Later among the works it cites.
CodeXGLUE: A machine learning benchmark dataset for code understanding and generation
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al · 2021
Later among the works it cites.
Codenet: A large-scale AI for code dataset for learning a diversity of coding tasks
R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, et al · 2021
Later among the works it cites.
CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation
Y. Wang, W. Wang, S. Joty, and S. C. Hoi · 2021
Later among the works it cites.
Break-it-fix-it: Unsupervised learning for program repair
M. Yasunaga and P. Liang · 2021
Later among the works it cites.
Review4repair: Code review aided automatic program repairing
F. Huq, M. Hasan, M. M. A. Haque, S. Mahbub, A. Iqbal, and T. Ahmed · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…