2023

On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

Mastropaolo, Antonio, Pascarella, Luca, Guglielmi, Emanuela et al.

Understand

Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding.

The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description.
While the usefulness of Copilot is evident, it is still unclear to what extent it is robust.
Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function.

Built on

R. Karampatsis and C. A. Sutton, “Maybe deep neural networks are the best choice for modeling source code,” CoRR , vol. abs/1903.05734, 2019. [Online]. Available: http://arxiv.org/abs/1903.05734
Original
1903
Earlier work this paper cites.
V. I. Levenshtein et al. , “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady , vol. 10, no. 8. Soviet Union, 1966, pp. 707–710
1966
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318
2002
Earlier work this paper cites.
T. Xie and J. Pei, “Mapo: Mining api usages from open source repositories,” ser. MSR ’06, 2006
2006
Earlier work this paper cites.
M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering , ser. ESEC/FSE 2009, 2009, pp. 213–222
2009
Earlier work this paper cites.
J. Kim, S. Lee, S. Hwang, and S. Kim, “Adding examples into java documents,” in 2009 IEEE/ACM International Conference on Automated Software Engineering , 2009, pp. 540–544
2009
Earlier work this paper cites.
S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” CoRR , vol. abs/2009.10297, 2020. [Online]. Available: https://arxiv.org/abs/2009.10297
Original
2009
Earlier work this paper cites.
R. Robbes and M. Lanza, “Improving code completion with program history,” Automated Software Engineering , vol. 17, no. 2, pp. 181–212, 2010
2010
Earlier work this paper cites.
A. Tamrawi, T. T. Nguyen, J. M. Al-Kofahi, and T. N. Nguyen, “Fuzzy set and cache-based approach for bug triaging,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ser. ESEC/FSE ’11, 2011, p. 365?375
2011
Earlier work this paper cites.
A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the naturalness of software,” in Proceedings of the 34th International Conference on Software Engineering , ser. ICSE 2012. IEEE Press, 2012, pp. 837–847
2012
Earlier work this paper cites.
C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each,” in 2012 34th International Conference on Software Engineering (ICSE) , 2012, pp. 3–13
2012
Earlier work this paper cites.
A. T. Nguyen, T. T. Nguyen, H. A. Nguyen, A. Tamrawi, H. V. Nguyen, J. Al-Kofahi, and T. N. Nguyen, “Graph-based pattern-oriented, context-sensitive source code completion,” in 2012 34th International Conference on Software Engineering (ICSE) , 2012, pp. 69–79
2012
Earlier work this paper cites.
C. McMillan, D. Poshyvanyk, M. Grechanik, Q. Xie, and C. Fu, “Portfolio: Searching for relevant functions and their usages in millions of lines of code,” ACM Trans. Softw. Eng. Methodol. , vol. 22, no. 4, pp. 37:1–37:30, 2013
2013
Earlier work this paper cites.
M. Allamanis, E. T. Barr, C. Bird, and C. Sutton, “Learning natural coding conventions,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2014, 2014, pp. 281–293
2014
Earlier work this paper cites.
M. Asaduzzaman, C. K. Roy, K. A. Schneider, and D. Hou, “Context-sensitive code completion tool for better api usability,” in 2014 IEEE International Conference on Software Maintenance and Evolution , 2014, pp. 621–624
2014
Earlier work this paper cites.
G. Bavota, A. D. Lucia, A. Marcus, and R. Oliveto, “Automating extract class refactoring: an improved method and its evaluation,” Empir. Softw. Eng. , vol. 19, no. 6, pp. 1617–1664, 2014
2014
Earlier work this paper cites.
M. P. Robillard, W. Maalej, R. J. Walker, and T. Zimmermann, Recommendation Systems in Software Engineering . Springer Publishing Company, Incorporated, 2014
2014
Earlier work this paper cites.
Z. Tu, Z. Su, and P. Devanbu, “On the localness of software,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2014. New York, NY, USA: Association for Computing Machinery, 2014, p. 269–280. [Online]. Available: https://doi.org/10.1145/2635868.2635875
2014
Earlier work this paper cites.
L. Moreno, G. Bavota, M. Di Penta, R. Oliveto, and A. Marcus, “How can i use this method?” in Proceedings of the 37th International Conference on Software Engineering - Volume 1 , ser. ICSE ’15, 2015, p. 880?890
2015
Earlier work this paper cites.
M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk, “Toward deep learning software repositories,” in Proceedings of the 12th Working Conference on Mining Software Repositories , ser. MSR ’15. Piscataway, NJ, USA: IEEE Press, 2015, pp. 334–345. [Online]. Available: http://dl.acm.org/citation.cfm?id=2820518.2820559
2015
Earlier work this paper cites.

Similar

A. T. Nguyen, H. A. Nguyen, and T. N. Nguyen, “A large-scale study on repetitiveness, containment, and composability of routines in open-source projects,” in Proceedings of the IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR 2016) , 2016, pp. 362–373
2016
Cited alongside, same era.
T. Nguyen, P. C. Rigby, A. T. Nguyen, M. Karanfil, and T. N. Nguyen, “T2api: Synthesizing api code usage templates from english texts with statistical translation,” in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering , ser. FSE 2016, 2016, p. 1013?1017
2016
Cited alongside, same era.
S. Proksch, S. Amann, S. Nadi, and M. Mezini, “Evaluating the evaluations of code recommender systems: a reality check,” in 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2016, pp. 111–121
2016
Cited alongside, same era.
F. V. Arrebola and P. T. A. Junior, “On source code completion assistants and the need of a context-aware approach,” in International Conference on Human Interface and the Management of Information . Springer, 2017, pp. 191–201
2017
Cited alongside, same era.
V. J. Hellendoorn and P. Devanbu, “Are deep neural networks the best choice for modeling source code?” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE 2017, 2017, p. 763?773
2017
Cited alongside, same era.
L. Moreno, G. Bavota, M. D. Penta, R. Oliveto, A. Marcus, and G. Canfora, “Arena: An approach for the automated generation of release notes,” IEEE Transactions on Software Engineering , vol. 43, no. 2, pp. 106–127, 2017
2017
Cited alongside, same era.
H. Niu, I. Keivanloo, and Y. Zou, “Api usage pattern recommendation for software development,” Journal of Systems and Software , vol. 129, pp. 127–139, 2017
2017
Cited alongside, same era.
X. Xia, D. Lo, Y. Ding, J. M. Al-Kofahi, T. N. Nguyen, and X. Wang, “Improving automated bug triaging with specialized topic model,” IEEE Transactions on Software Engineering , vol. 43, no. 3, pp. 272–297, 2017
2017
Cited alongside, same era.
G. A. Campbell, “Cognitive complexity: An overview and evaluation,” in Proceedings of the 2018 international conference on technical debt , 2018, pp. 57–58
2018
Cited alongside, same era.
X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in Proceedings of the 26th Conference on Program Comprehension, ICPC 2018, Gothenburg, Sweden, May 27-28, 2018 , F. Khomh, C. K. Roy, and J. Siegmund, Eds. ACM, 2018, pp. 200–210
2018
Cited alongside, same era.
——, “Deep code comment generation,” ser. ICPC ’18, 2018
2018
Cited alongside, same era.
X. Jin and F. Servant, “The hidden cost of code completion: Understanding the impact of the recommendation-list length on its efficiency,” in Proceedings of the 15th International Conference on Mining Software Repositories , 2018, pp. 70–73
2018
Cited alongside, same era.

Then

Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , ser. ICSE ’20, 2020, p. 602?614
2020
Later among the works it cites.
F. Liu, G. Li, Y. Zhao, and Z. Jin, “Multi-task learning based pre-trained language model for code completion,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE 2020. Association for Computing Machinery, 2020
2020
Later among the works it cites.
A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” arXiv preprint arXiv:2005.08025 , 2020
Original
2020
Later among the works it cites.
M. Tufano, D. Drain, A. Svyatkovskiy, and N. Sundaresan, “Generating accurate assert statements for unit test cases using pretrained transformers,” CoRR , vol. abs/2009.05634, 2020
Original
2020
Later among the works it cites.
C. Watson, M. Tufano, K. Moran, G. Bavota, and D. Poshyvanyk, “On learning meaningful assert statements for unit test cases,” in Proceedings of the 42nd International Conference on Software Engineering, ICSE 2020 , 2020, p. To Appear
2020
Later among the works it cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Later among the works it cites.
M. Ciniselli, N. Cooper, L. Pascarella, D. Poshyvanyk, M. Di Penta, and G. Bavota, “An empirical study on the usage of bert models for code completion,” in Proceedings of the 18th Working Conference on Mining Software Repositories , ser. MSR ’21, 2021, p. To Appear
2021
Later among the works it cites.
O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for msr studies,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . IEEE, 2021, pp. 560–564
2021
Later among the works it cites.
G. D. Howard, “Github copilot: Copyright, fair use, creativity, transformativity, and algorithms,” 2021
2021
Later among the works it cites.
J. Li, R. Huang, W. Li, K. Yao, and W. Tan, “Toward less hidden cost of code completion with acceptance and ranking models,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2021, pp. 195–205
2021
Later among the works it cites.
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “An empirical cybersecurity evaluation of github copilot’s code contributions,” arXiv preprint arXiv:2108.09293 , 2021
Original
2021
Later among the works it cites.
D. Sobania, M. Briesch, and F. Rothlauf, “Choose your programming copilot: A comparison of the program synthesis performance of github copilot and genetic programming,” arXiv preprint arXiv:2111.07875 , 2021
Original
2021
Later among the works it cites.
F. Wen, E. Aghajani, C. Nagy, M. Lanza, and G. Bavota, “Siri, write the next method,” in 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021 . IEEE, 2021, pp. 138–149
2021
Later among the works it cites.
F. F. Xu, B. Vasilescu, and G. Neubig, “In-ide code generation from natural language: Promise and challenges,” 2021
2021
Later among the works it cites.
N. A. Ernst and G. Bavota, “Ai-driven development is here: Should you worry?” IEEE Softw. , vol. 39, no. 2, pp. 106–110, 2022
2022
Later among the works it cites.
S. Imai, “Is github copilot a substitute for human pair-programming? an empirical study,” in 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 2022, pp. 319–321
2022
Later among the works it cites.
N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) . IEEE, 2022, pp. 1–5
2022
Later among the works it cites.
P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in CHI Conference on Human Factors in Computing Systems Extended Abstracts , 2022, pp. 1–7
2022
Later among the works it cites.
A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022, pp. 21–29
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…