2023

Generation-based Code Review Automation: How Far Are We?

Zhou, Xin, Kim, Kisub, Xu, Bowen et al.

Understand

Code review is an effective software quality assurance activity; however, it is labor-intensive and time-consuming.

Thus, a number of generation-based automatic code review (ACR) approaches have been proposed recently, which leverage deep learning techniques to automate various activities in the code review process (e.g., code revision generation and review comment generation).
We find the previous works carry three main limitations.
First, the ACR approaches have been shown to be beneficial in each work, but those methods are not comprehensively compared with each other to show their superiority over their peer ACR approaches.

Built on

H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” CoRR , vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436
Original
1909
Earlier work this paper cites.
V. I. Levenshtein et al. , “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady , vol. 10, no. 8. Soviet Union, 1966, pp. 707–710
1966
Earlier work this paper cites.
M. V. Zelkowitz, A. C. Shaw, and J. D. Gannon, Principles of software engineering and design . Prentice Hall Professional Technical Reference, 1979
1979
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Annual Meeting of the Association for Computational Linguistics , 2002
2002
Earlier work this paper cites.
J. P. Turian, L. Shea, and I. D. Melamed, “Evaluation of machine translation and its evaluation,” in Machine Translation Summit , 2003
2003
Earlier work this paper cites.
D. A. Coughlin, “Correlating automated and human assessments of machine translation quality,” in Machine Translation Summit , 2003
2003
Earlier work this paper cites.
A. J. Ko, B. A. Myers, M. J. Coblenz, and H. H. Aung, “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks,” IEEE Transactions on software engineering , vol. 32, no. 12, pp. 971–987, 2006
2006
Earlier work this paper cites.
G. J. Myers, C. Sandler, and T. Badgett, The art of software testing . John Wiley & Sons, 2011
2011
Earlier work this paper cites.
Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” IEEE Transactions on Software Engineering , vol. 39, no. 6, pp. 757–773, 2012
2012
Earlier work this paper cites.
R. K. Saha, M. Lease, S. Khurshid, and D. E. Perry, “Improving bug localization using structured information retrieval,” in 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2013, pp. 345–355
2013
Earlier work this paper cites.
A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in 2013 35th International Conference on Software Engineering (ICSE) . IEEE, 2013, pp. 712–721
2013
Earlier work this paper cites.
P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th joint meeting on foundations of software engineering , 2013, pp. 202–212
2013
Earlier work this paper cites.
S. Wang and D. Lo, “Version history, similar report, and structure: Putting them together for improved bug localization,” in Proceedings of the 22nd International Conference on Program Comprehension , 2014, pp. 53–63
2014
Earlier work this paper cites.
S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,” in Proceedings of the 11th working conference on mining software repositories , 2014, pp. 192–201
2014
Earlier work this paper cites.
M. Beller, A. Bacchelli, A. Zaidman, and E. Juergens, “Modern code reviews in open-source projects: Which problems do they fix?” in Proceedings of the 11th working conference on mining software repositories , 2014, pp. 202–211
2014
Earlier work this paper cites.
P. C. Rigby, D. M. German, L. Cowen, and M.-A. Storey, “Peer review on open-source software projects: Parameters, statistical models, and theory,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 23, no. 4, pp. 1–33, 2014
2014
Earlier work this paper cites.
G. Bavota and B. Russo, “Four eyes are better than two: On the impact of code reviews on software quality,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2015, pp. 81–90
2015
Earlier work this paper cites.
R. Morales, S. McIntosh, and F. Khomh, “Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,” in 2015 IEEE 22nd international conference on software analysis, evolution, and reengineering (SANER) . IEEE, 2015, pp. 171–180
2015
Earlier work this paper cites.
R. Minelli, A. Mocci, and M. Lanza, “I know what you did last summer-an investigation of how developers spend their time,” in 2015 IEEE 23rd International Conference on Program Comprehension . IEEE, 2015, pp. 25–35
2015
Earlier work this paper cites.
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909 , 2015
Original
2015
Earlier work this paper cites.
P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo, “Understanding the test automation culture of app developers,” in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) . IEEE, 2015, pp. 1–10
2015
Earlier work this paper cites.
P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, and K.-i. Matsumoto, “Who should review my code? a file location-based code-reviewer recommendation approach for modern code review,” in 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) . IEEE, 2015, pp. 141–150
2015
Earlier work this paper cites.
S. Wang, T. Liu, and L. Tan, “Automatically learning semantic features for defect prediction,” in Proceedings of the 38th International Conference on Software Engineering , 2016, pp. 297–308
2016
Earlier work this paper cites.
V. Garousi and M. V. Mäntylä, “When and what to automate in software testing? a multi-vocal literature review,” Information and Software Technology , vol. 76, pp. 92–117, 2016
2016
Earlier work this paper cites.

Similar

A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chockley, “Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,” IEEE Transactions on Software Engineering , vol. 43, no. 1, pp. 56–75, 2016
2016
Cited alongside, same era.
S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,” Empirical Software Engineering , vol. 21, no. 5, pp. 2146–2189, 2016
2016
Cited alongside, same era.
P. S. Kochhar, X. Xia, D. Lo, and S. Li, “Practitioners’ expectations on automated fault localization,” in Proceedings of the 25th International Symposium on Software Testing and Analysis , 2016, pp. 165–176
2016
Cited alongside, same era.
M. M. Rahman, C. K. Roy, J. Redl, and J. A. Collins, “Correct: Code reviewer recommendation at github for vendasta technologies,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering , 2016, pp. 792–797
2016
Cited alongside, same era.
X. Xia, L. Bao, D. Lo, Z. Xing, A. E. Hassan, and S. Li, “Measuring program comprehension: A large-scale field study with professionals,” IEEE Transactions on Software Engineering , vol. 44, no. 10, pp. 951–976, 2017
2017
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
2017
Cited alongside, same era.
V. J. Hellendoorn and P. Devanbu, “Are deep neural networks the best choice for modeling source code?” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , 2017, pp. 763–773
2017
Cited alongside, same era.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017
Original
2017
Cited alongside, same era.
C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” in Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice , 2018, pp. 181–190
2018
Cited alongside, same era.
D. Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre, “Learning long-range spatial dependencies with horizontal gated recurrent units,” Advances in neural information processing systems , vol. 31, 2018
2018
Cited alongside, same era.
Y. Fan, X. Xia, D. Lo, and S. Li, “Early prediction of merged code changes to prioritize reviewing tasks,” Empirical Software Engineering , vol. 23, no. 6, pp. 3346–3393, 2018
2018
Cited alongside, same era.
S. McIntosh and Y. Kamei, “Are fix-inducing changes a moving target? a longitudinal case study of just-in-time defect prediction,” in Proceedings of the 40th International Conference on Software Engineering , 2018, pp. 560–560
2018
Cited alongside, same era.

Then

W. Ma, Y. Cui, C. Si, T. Liu, S. Wang, and G. Hu, “Charbert: character-aware pre-trained language model,” arXiv preprint arXiv:2011.01513 , 2020
Original
2020
Later among the works it cites.
J. K. Siow, C. Gao, L. Fan, S. Chen, and Y. Liu, “Core: Automating review recommendation for code changes,” in 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2020, pp. 284–295
2020
Later among the works it cites.
W. H. A. Al-Zubaidi, P. Thongtanunam, H. K. Dam, C. Tantithamthavorn, and A. Ghose, “Workload-aware reviewer recommendation using a multi-objective search-based approach,” in Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering , 2020, pp. 21–30
2020
Later among the works it cites.
T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: distributed representations of code changes,” in ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020 , G. Rothermel and D. Bae, Eds. ACM, 2020, pp. 518–529. [Online]. Available: https://doi.org/10.1145/3377811.3380361
2020
Later among the works it cites.
R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 163–174
2021
Later among the works it cites.
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=jLoC4ez43PZ
2021
Later among the works it cites.
Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , M. Moens, X. Huang, L. Specia, and S. W. Yih, Eds. Association for Computational Linguistics, 2021, pp. 8696–8708. [Online]. Available: https://doi.org/10.18653/v1/2021.emnlp-main.685
2021
Later among the works it cites.
X. Zhou, D. Han, and D. Lo, “Assessing generalizability of codebert,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2021, pp. 425–436
2021
Later among the works it cites.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021
Original
2021
Later among the works it cites.
A. Elgohary, C. Meek, M. Richardson, A. Fourney, G. Ramos, and A. H. Awadallah, “NL-EDIT: Correcting semantic parse errors through natural language interaction,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Online: Association for Computational Linguistics, Jun. 2021, pp. 5599–5610. [Online]. Available: https://aclanthology.org/2021.naacl-main.444
2021
Later among the works it cites.
S. J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W. Y. Lee, B. Sagot et al. , “Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp,” arXiv preprint arXiv:2112.10508 , 2021
Original
2021
Later among the works it cites.
V. J. Hellendoorn, J. Tsay, M. Mukherjee, and M. Hirzel, “Towards automating code review at scale,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 1479–1482
2021
Later among the works it cites.
C. Pornprasit and C. K. Tantithamthavorn, “Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . IEEE, 2021, pp. 369–379
2021
Later among the works it cites.
P. Thongtanunam, C. Pornprasit, and C. Tantithamthavorn, “Autotransform: Automated code transformation to support modern code review process,” in 2022 IEEE/ACM International Conference on Software Engineering (ICSE) , 2022
2022
Later among the works it cites.
R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automation,” in 2022 IEEE/ACM International Conference on Software Engineering (ICSE) , 2022
2022
Later among the works it cites.
E. Winter, D. Bowes, S. Counsell, T. Hall, S. Haraldsson, V. Nowack, and J. Woodward, “How do developers really feel about bug fixing? directions for automatic program repair,” IEEE Transactions on Software Engineering , 2022
2022
Later among the works it cites.
V. Dibia, A. Fourney, G. Bansal, F. Poursabzi-Sangdeh, H. Liu, and S. Amershi, “Aligning offline metrics and human judgments of value of ai-pair programmers,” arXiv preprint arXiv:2210.16494 , 2022
Original
2022
Later among the works it cites.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311 , 2022
Original
2022
Later among the works it cites.
L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel, “Byt5: Towards a token-free future with pre-trained byte-to-byte models,” Transactions of the Association for Computational Linguistics , vol. 10, pp. 291–306, 2022
2022
Later among the works it cites.
Y. Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Commentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 507–519
2022
Later among the works it cites.
Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al. , “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1035–1047
2022
Later among the works it cites.
B. Wu, B. Liang, and X. Zhang, “Turn tree into graph: Automatic code review via simplified ast driven graph convolutional network,” Knowledge-Based Systems , vol. 252, p. 109450, 2022
2022
Later among the works it cites.
X. Zhou, D. Han, and D. Lo, “Simple or complex? together for a more accurate just-in-time defect predictor,” in 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC) . IEEE, 2022, pp. 229–240
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…