2023

Impact of Code Language Models on Automated Program Repair

Jiang, Nan, Liu, Kevin, Lutellier, Thibaud et al.

Understand

Automated program repair (APR) aims to help developers improve software reliability by generating patches for buggy programs.

Although many code language models (CLM) are developed and effective in many software tasks such as code completion, there has been little comprehensive, in-depth work to evaluate CLMs' fixing capabilities and to fine-tune CLMs for the APR task.
Firstly, this work is the first to evaluate ten CLMs on four APR benchmarks, which shows that surprisingly, the best CLM, as is, fixes 72% more bugs than the state-of-the-art deep-learning (DL)-based APR techniques.
Secondly, one of the four APR benchmarks was created by us in this paper to avoid data leaking for a fair evaluation.

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
Original
1907
Earlier work this paper cites.
H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” CoRR , vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436
Original
1909
Earlier work this paper cites.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” CoRR , vol. abs/1910.13461, 2019. [Online]. Available: http://arxiv.org/abs/1910.13461
Original
1910
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” CoRR , vol. abs/1910.10683, 2019. [Online]. Available: http://arxiv.org/abs/1910.10683
Original
1910
Earlier work this paper cites.
J. Zhang, H. Zhang, C. Xia, and L. Sun, “Graph-bert: Only attention is needed for learning graph representations,” CoRR , vol. abs/2001.05140, 2020. [Online]. Available: https://arxiv.org/abs/2001.05140
Original
2001
Earlier work this paper cites.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” CoRR , vol. abs/2002.08155, 2020. [Online]. Available: https://arxiv.org/abs/2002.08155
Original
2002
Earlier work this paper cites.
C.-Y. Lin and F. J. Och, “ORANGE: a method for evaluating automatic evaluation metrics for machine translation,” in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics . Geneva, Switzerland: COLING, aug 23–aug 27 2004, pp. 501–507. [Online]. Available: https://aclanthology.org/C04-1072
2004
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR , vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” CoRR , vol. abs/2009.08366, 2020. [Online]. Available: https://arxiv.org/abs/2009.08366
Original
2009
Earlier work this paper cites.
V. P. Dwivedi and X. Bresson, “A generalization of transformer networks to graphs,” CoRR , vol. abs/2012.09699, 2020. [Online]. Available: https://arxiv.org/abs/2012.09699
Original
2012
Earlier work this paper cites.
R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs,” in ISSTA , 2014, pp. 437–440
2014
Earlier work this paper cites.
M. Tan, L. Tan, S. Dara, and C. Mayeux, “Online defect prediction for imbalanced data,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 2. IEEE, 2015, pp. 99–108
2015
Earlier work this paper cites.
J. Xuan, M. Martinez, F. Demarco, M. Clement, S. R. L. Marcote, T. Durieux, D. L. Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,” IEEE Trans. Software Eng. , vol. 43, no. 1, pp. 34–55, 2017. [Online]. Available: https://doi.org/10.1109/TSE.2016.2560811
2016
Earlier work this paper cites.
Z. Hu, X. Ma, Z. Liu, E. Hovy, and E. Xing, “Harnessing deep neural networks with logic rules,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 2410–2420. [Online]. Available: https://aclanthology.org/P16-1228
2016
Earlier work this paper cites.
R. K. Saha, Y. Lyu, H. Yoshida, and M. R. Prasad, “ELIXIR: effective object oriented program repair,” in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017 , G. Rosu, M. D. Penta, and T. N. Nguyen, Eds. IEEE Computer Society, 2017, pp. 648–659. [Online]. Available: https://doi.org/10.1109/ASE.2017.8115675
2017
Earlier work this paper cites.
Y. Xiong, J. Wang, R. Yan, J. Zhang, S. Han, G. Huang, and L. Zhang, “Precise condition synthesis for program repair,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) , 2017, pp. 416–426
2017
Earlier work this paper cites.
D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama, “QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge,” in SPLASH , 2017, p. 55–56
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” CoRR , vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
Original
2017
Earlier work this paper cites.
R. Gupta, S. Pal, A. Kanade, and S. Shevade, “Deepfix: Fixing Common C Language Errors by Deep Learning,” in AAAI , 2017, pp. 1345–1351
2017
Earlier work this paper cites.
E. A. Santos, J. C. Campbell, A. Hindle, and J. N. Amaral, “Finding and Correcting Syntax Errors Using Recurrent Neural Networks,” PeerJ PrePrints , vol. 5, p. e3123v1, 2017
2017
Earlier work this paper cites.
M. Monperrus, “Automatic software repair: A bibliography,” ACM Comput. Surv. , vol. 51, no. 1, pp. 17:1–17:24, 2018. [Online]. Available: https://doi.org/10.1145/3105906
2018
Earlier work this paper cites.

Similar

J. Jiang, Y. Xiong, H. Zhang, Q. Gao, and X. Chen, “Shaping program repair space with existing patches and similar code,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 298–309. [Online]. Available: https://doi.org/10.1145/3213846.3213871
2018
Cited alongside, same era.
Y. Yuan and W. Banzhaf, “ARJA: automated repair of java programs via multi-objective genetic programming,” IEEE Trans. Software Eng. , vol. 46, no. 10, pp. 1040–1067, 2020. [Online]. Available: https://doi.org/10.1109/TSE.2018.2874648
2018
Cited alongside, same era.
M. Wen, J. Chen, R. Wu, D. Hao, and S. Cheung, “Context-aware patch generation for better automated program repair,” in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018 , M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, pp. 1–11. [Online]. Available: https://doi.org/10.1145/3180155.3180233
2018
Cited alongside, same era.
M. Martinez and M. Monperrus, “Ultra-large repair search space with automatically mined templates: The cardumen mode of astor,” in Search-Based Software Engineering - 10th International Symposium, SSBSE 2018, Montpellier, France, September 8-9, 2018, Proceedings , ser. Lecture Notes in Computer Science, T. E. Colanzi and P. McMinn, Eds., vol. 11036. Springer, 2018, pp. 65–86. [Online]. Available: https://doi.org/10.1007/978-3-319-99241-9_3
2018
Cited alongside, same era.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018
2018
Cited alongside, same era.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
Original
2018
Cited alongside, same era.
C. L. Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” Commun. ACM , vol. 62, no. 12, pp. 56–65, 2019. [Online]. Available: https://doi.org/10.1145/3318162
2019
Cited alongside, same era.
Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair,” TSE , 2019
2019
Cited alongside, same era.
K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “TBar: Revisiting Template-Based Automated Program Repair,” in ISSTA . ACM, 2019
2019
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Cited alongside, same era.
M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in Proceedings of NAACL-HLT 2019: Demonstrations , 2019
2019
Cited alongside, same era.
A. Mesbah, A. Rice, E. Johnston, N. Glorioso, and E. Aftandilian, “DeepDelta: Learning to Repair Compilation Errors,” in ESEC/FSE . ACM, 2019, pp. 925–936
2019
Cited alongside, same era.

Then

B. Wang and A. Komatsuzaki, “GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model,” May 2021
2021
Later among the works it cites.
S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman, “GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow,” Mar. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5297715
2021
Later among the works it cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” CoRR , vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374
Original
2021
Later among the works it cites.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” CoRR , vol. abs/2102.04664, 2021
Original
2021
Later among the works it cites.
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” CoRR , vol. abs/2101.00027, 2021. [Online]. Available: https://arxiv.org/abs/2101.00027
Original
2021
Later among the works it cites.
X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. S. Koura, V. Chaudhary, B. O’Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. T. Diab, V. Stoyanov, and X. Li, “Few-shot learning with multilingual language models,” CoRR , vol. abs/2112.10668, 2021. [Online]. Available: https://arxiv.org/abs/2112.10668
Original
2021
Later among the works it cites.
S. Chakraborty, Y. Ding, M. Allamanis, and B. Ray, “Codit: Code editing with tree-based neural models,” IEEE Transactions on Software Engineering , vol. 48, no. 4, pp. 1385–1399, 2022
2022
Later among the works it cites.
H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” in Proceedings of the International Conference on Software Engineering , 2022. [Online]. Available: http://arxiv.org/pdf/2105.04123
Original
2022
Later among the works it cites.
Z. Chen, S. Kommrusch, and M. Monperrus, “Neural transfer learning for repairing security vulnerabilities in c code,” IEEE Transactions on Software Engineering , 2022. [Online]. Available: http://arxiv.org/pdf/2104.08308
Original
2022
Later among the works it cites.
H. Ye, M. Martinez, X. Luo, T. Zhang, and M. Monperrus, “Selfapr: Self-supervised program repair with test execution diagnostics,” in Proceedings of ASE , 2022. [Online]. Available: http://arxiv.org/pdf/2203.12755
Original
2022
Later among the works it cites.
Y. Noller, R. Shariffdeen, X. Gao, and A. Roychoudhury, “Trust enhancement issues in program repair,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2228–2240. [Online]. Available: https://doi.org/10.1145/3510003.3510040
2022
Later among the works it cites.
S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, E. Zheng, R. Child, R. Y. Aminabadi, J. Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro, “Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model,” CoRR , vol. abs/2201.11990, 2022. [Online]. Available: https://arxiv.org/abs/2201.11990
Original
2022
Later among the works it cites.
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A conversational paradigm for program synthesis,” arXiv preprint , 2022
2022
Later among the works it cites.
J. A. Prenner, H. Babii, and R. Robbes, “Can openai’s codex fix bugs?: An evaluation on quixbugs,” in 2022 IEEE/ACM International Workshop on Automated Program Repair (APR) , 2022, pp. 69–75
2022
Later among the works it cites.
D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” 2022. [Online]. Available: https://arxiv.org/abs/2204.05999
Original
2022
Later among the works it cites.
E.-S. A. Lee, S. Thillainathan, S. Nayak, S. Ranathunga, D. I. Adelani, R. Su, and A. D. McCarthy, “Pre-trained multilingual sequence-to-sequence models: A hope for low-resource language translation?” 2022. [Online]. Available: https://arxiv.org/abs/2203.08850
Original
2022
Later among the works it cites.
H. Mehrafarin, S. Rajaee, and M. T. Pilehvar, “On the importance of data size in probing fine-tuned models,” in Findings of the Association for Computational Linguistics: ACL 2022 . Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 228–238. [Online]. Available: https://aclanthology.org/2022.findings-acl.20
2022
Later among the works it cites.
C. S. Xia and L. Zhang, “Less training, more repairing please: Revisiting automated program repair via zero-shot learning,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 959–971. [Online]. Available: https://doi.org/10.1145/3540250.3549101
2022
Later among the works it cites.
“Replication package of this work,” 2022. [Online]. Available: https://github.com/lin-tan/clm
2022
Later among the works it cites.
N. Jiang, T. Lutellier, Y. Lou, L. Tan, D. Goldwasser, and X. Zhang, “Knod: Domain knowledge distilled tree decoder for automated program repair,” in Proceedings of the International Conference on Software Engineering , 2023
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…