2024

Aligning the Objective of LLM-based Program Repair

Xu, Junjielong, Fu, Ying, Tan, Shin Hwei et al.

Understand

Large language models (LLMs) have achieved decent results on automated program repair (APR).

However, the next token prediction training objective of decoder-only LLMs (e.g., GPT-4) is misaligned with the masked span prediction objective of current infilling-style methods, which impedes LLMs from fully leveraging pre-trained knowledge for program repair.
In addition, while some LLMs can locate and repair bugs in certain functions using the related artifacts (e.g., test cases), existing methods still depend on statement-level fault localization methods to provide a list of buggy hunks for repair.
This restriction hinders LLMs from exploring potential patches beyond the given locations.

Built on

C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,” Ieee transactions on software engineering , vol. 38, no. 1, pp. 54–72, 2011
2011
Earlier work this paper cites.
H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra, “Semfix: Program repair via semantic analysis,” in 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 2013, pp. 772–781
2013
Earlier work this paper cites.
D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” in 2013 35th international conference on software engineering (ICSE) . IEEE, 2013, pp. 802–811
2013
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” Advances in neural information processing systems , vol. 27, 2014
2014
Earlier work this paper cites.
R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Proceedings of the 2014 international symposium on software testing and analysis , 2014, pp. 437–440
2014
Earlier work this paper cites.
Z. Tu, Z. Su, and P. Devanbu, “On the localness of software,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2014, pp. 269–280
2014
Earlier work this paper cites.
F. Long and M. Rinard, “Staged program repair with condition synthesis,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, pp. 166–178
2015
Earlier work this paper cites.
S. H. Tan and A. Roychoudhury, “relifix: Automated repair of software regressions,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1. IEEE, 2015, pp. 471–482
2015
Earlier work this paper cites.
E. K. Smith, E. T. Barr, C. Le Goues, and Y. Brun, “Is the cure worse than the disease? overfitting in automated program repair,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, pp. 532–543
2015
Earlier work this paper cites.
Z. Qi, F. Long, S. Achour, and M. Rinard, “An analysis of patch plausibility and correctness for generate-and-validate patch generation systems,” in Proceedings of the 2015 International Symposium on Software Testing and Analysis , 2015, pp. 24–36
2015
Earlier work this paper cites.
S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” in Proceedings of the 38th international conference on software engineering , 2016, pp. 691–701
2016
Earlier work this paper cites.
W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” IEEE Transactions on Software Engineering , vol. 42, no. 8, pp. 707–740, 2016
2016
Earlier work this paper cites.
X. B. D. Le, D. Lo, and C. Le Goues, “History driven program repair,” in 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER) , vol. 1. IEEE, 2016, pp. 213–224
2016
Earlier work this paper cites.
J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,” IEEE Transactions on Software Engineering , vol. 43, no. 1, pp. 34–55, 2016
2016
Earlier work this paper cites.
A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the naturalness of software,” Communications of the ACM , vol. 59, no. 5, pp. 122–131, 2016
2016
Earlier work this paper cites.
S. H. Tan, H. Yoshida, M. R. Prasad, and A. Roychoudhury, “Anti-patterns in search-based program repair,” in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering , 2016, pp. 727–738
2016
Earlier work this paper cites.
D. Yang, Y. Qi, and X. Mao, “An empirical study on the usage of fault localization in automated program repair,” in 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2017, pp. 504–508
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017
2017
Earlier work this paper cites.
J. Sohn and S. Yoo, “Fluccs: Using code and change metrics to improve fault localization,” in Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2017, pp. 273–283
2017
Earlier work this paper cites.
M. Monperrus, “Automatic software repair: A bibliography,” ACM Computing Surveys (CSUR) , vol. 51, no. 1, pp. 1–24, 2018
2018
Earlier work this paper cites.
M. Wen, J. Chen, R. Wu, D. Hao, and S.-C. Cheung, “Context-aware patch generation for better automated program repair,” in Proceedings of the 40th international conference on software engineering , 2018, pp. 1–11
2018
Earlier work this paper cites.
J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Sketchfix: a tool for automated program repair approach using lazy candidate generation,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2018, pp. 888–891
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
G. An, J. Kim, and S. Yoo, “Comparing line and ast granularity level for program repair using pyggi,” in Proceedings of the 4th International Workshop on Genetic Improvement Workshop , 2018, pp. 19–26
2018
Earlier work this paper cites.

Similar

Y. Yuan and W. Banzhaf, “Arja: Automated repair of java programs via multi-objective genetic programming,” IEEE Transactions on software engineering , vol. 46, no. 10, pp. 1040–1067, 2018
2018
Cited alongside, same era.
J. Jiang, Y. Xiong, H. Zhang, Q. Gao, and X. Chen, “Shaping program repair space with existing patches and similar code,” in Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis , 2018, pp. 298–309
2018
Cited alongside, same era.
K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Tbar: Revisiting template-based automated program repair,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2019, pp. 31–42
2019
Cited alongside, same era.
Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end-to-end program repair,” IEEE Transactions on Software Engineering , vol. 47, no. 9, pp. 1943–1959, 2019
2019
Cited alongside, same era.
K. Liu, A. Koyuncu, T. F. Bissyandé, D. Kim, J. Klein, and Y. Le Traon, “You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems,” in 2019 12th IEEE conference on software testing, validation and verification (ICST) . IEEE, 2019, pp. 102–113
2019
Cited alongside, same era.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020
2020
Cited alongside, same era.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res. , vol. 21, no. 1, jan 2020
2020
Cited alongside, same era.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020
Original
2020
Cited alongside, same era.
Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 341–353
2021
Cited alongside, same era.
R. Zhong, K. Lee, Z. Zhang, and D. Klein, “Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections,” arXiv preprint arXiv:2104.04670 , 2021
Original
2021
Cited alongside, same era.
Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021
Original
2021
Cited alongside, same era.
N. Jiang, T. Lutellier, and L. Tan, “Cure: Code-aware neural machine translation for automatic program repair,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 1161–1173
2021
Cited alongside, same era.

Then

C. S. Xia, Y. Ding, and L. Zhang, “Revisiting the plastic surgery hypothesis via large language models,” arXiv preprint arXiv:2303.10494 , 2023
Original
2023
Later among the works it cites.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023
Original
2023
Later among the works it cites.
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712 , 2023
Original
2023
Later among the works it cites.
C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery , 2023
2023
Later among the works it cites.
C. S. Xia and L. Zhang, “Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” arXiv preprint arXiv:2304.00385 , 2023
Original
2023
Later among the works it cites.
——, “Conversational automated program repair,” arXiv preprint arXiv:2301.13246 , 2023
Original
2023
Later among the works it cites.
D. Cheng, S. Huang, and F. Wei, “Adapting large language models via reading comprehension,” arXiv preprint arXiv:2309.09530 , 2023
Original
2023
Later among the works it cites.
D. Huang, Q. Bu, J. M. Zhang, M. Luck, and H. Cui, “Agentcoder: Multi-agent-based code generation with iterative testing and optimisation,” arXiv preprint arXiv:2312.13010 , 2023
Original
2023
Later among the works it cites.
X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug,” arXiv preprint arXiv:2304.05128 , 2023
Original
2023
Later among the works it cites.
Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated repair of programs from large language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1469–1481
2023
Later among the works it cites.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023
Original
2023
Later among the works it cites.
M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2023, pp. 1646–1656
2023
Later among the works it cites.
T. Ahmed and P. Devanbu, “Better patching using llm prompting, via self-consistency,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 1742–1746
2023
Later among the works it cites.
J. Xu, Z. Cui, Y. Zhao, X. Zhang, S. He, P. He, L. Li, Y. Kang, Q. Lin, Y. Dang et al. , “Unilog: Automatic logging via llm and in-context learning,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 2023, pp. 129–140
2023
Later among the works it cites.
Y. Wei, C. S. Xia, and L. Zhang, “Copiloting the copilots: Fusing large language models with completion engines for automated program repair,” arXiv preprint arXiv:2309.00608 , 2023
Original
2023
Later among the works it cites.
Y. Wan, Y. Dong, J. Xiao, Y. Huo, W. Wang, and M. R. Lyu, “Mrweb: An exploration of generating multi-page resource-aware web code from ui designs,” arXiv preprint arXiv:2412.15310 , 2024
Original
2024
Closest in time.
W. Wang, Y. Su, J. Huan, J. Liu, W. Chen, Y. Zhang, C.-Y. Li, K.-J. Chang, X. Xin, L. Shen et al. , “Asclepius: A spectrum evaluation benchmark for medical multi-modal large language models,” arXiv preprint arXiv:2402.11217 , 2024
Original
2024
Closest in time.
C. Lee, C. S. Xia, L. Yang, J.-t. Huang, Z. Zhu, L. Zhang, and M. R. Lyu, “A unified debugging approach via llm-based multi-agent synergy,” arXiv preprint arXiv:2404.17153 , 2024
Original
2024
Closest in time.
R. Tian, Y. Ye, Y. Qin, X. Cong, Y. Lin, Z. Liu, and M. Sun, “Debugbench: Evaluating debugging capability of large language models,” arXiv preprint arXiv:2401.04621 , 2024
Original
2024
Closest in time.
J. Xu, R. Yang, Y. Huo, C. Zhang, and P. He, “Divlog: Log parsing with prompt enhanced in-context learning,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–12
2024
Closest in time.
X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou, “Evaluating large language models in class-level code generation,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 2024, pp. 865–865
2024
Closest in time.
J. Cao, Y.-K. Chan, Z. Ling, W. Wang, S. Li, M. Liu, R. Qiao, Y. Han, C. Wang, B. Yu et al. , “How should i build a benchmark? revisiting code-related benchmarks for llms,” arXiv e-prints , pp. arXiv–2501, 2025
2025
Closest in time.
W. Wang, Z. Ma, Z. Wang, C. Wu, W. Chen, X. Li, and Y. Yuan, “A survey of llm-based agents in medicine: How far are we from baymax?” arXiv preprint arXiv:2502.11211 , 2025
Original
2025
Closest in time.
C. Li, T. Xu, and Y. Guo, “Reasoning-as-logic-units: Scaling test-time reasoning in large language models through logic unit alignment,” arXiv preprint arXiv:2502.07803 , 2025
Original
2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…