2022

Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?

Asare, Owura, Nagappan, Meiyappan, Asokan, N.

Understand

Several advances in deep learning have been successfully applied to the software development process.

Of recent interest is the use of neural language models to build tools, such as Copilot, that assist in writing code.
In this paper we perform a comparative empirical analysis of Copilot-generated code from a security perspective.
The aim of this study is to determine if Copilot is as bad as human developers.

Built on

Long Short-Term Memory
Hochreiter, S. and J. Schmidhuber. 1997, November · 1997
Earlier work this paper cites.
A Neural Probabilistic Language Model
Bengio, Y., R. Ducharme, and P. Vincent 2000 · 2000
Earlier work this paper cites.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Original
Feng, Z., D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. 2020, September · 2002
Earlier work this paper cites.
Language Models are Few-Shot Learners
Original
Brown, T.B., B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D.M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. 2020, July · 2005
Earlier work this paper cites.
On the Naturalness of Software
Hindle, A., E.T. Barr, Z. Su, M. Gabel, and P. Devanbu 2012 · 2012
Earlier work this paper cites.
A Fast and Accurate Dependency Parser using Neural Networks
Chen, D. and C. Manning 2014, October · 2014
Earlier work this paper cites.
Code completion with statistical language models
Raychev, V., M. Vechev, and E. Yahav 2014, June · 2014
Earlier work this paper cites.
PHOG: probabilistic model for code
Bielik, P., V. Raychev, and M. Vechev 2016 · 2016
Earlier work this paper cites.
A Neural Model for Part-of-Speech Tagging in Historical Texts
Hardmeier, C. 2016, December · 2016
Earlier work this paper cites.
Neural Generative Question Answering
Yin, J., X. Jiang, Z. Lu, L. Shang, H. Li, and X. Li 2016 · 2016
Earlier work this paper cites.
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
Zhou, J., Y. Cao, X. Wang, P. Li, and W. Xu. 2016 · 2016
Earlier work this paper cites.
Are deep neural networks the best choice for modeling source code?
Hellendoorn, V.J. and P. Devanbu 2017, August · 2017
Earlier work this paper cites.
Attention is All You Need
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, \. Kaiser, and I. Polosukhin 2017 · 2017
Earlier work this paper cites.
A Syntactic Neural Model for General-Purpose Code Generation
Original
Yin, P. and G. Neubig. 2017, April · 2017
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Devlin, J., M.W. Chang, K. Lee, and K. Toutanova. 2019, May · 2019
Earlier work this paper cites.

Similar

A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries
Fan, J., Y. Li, S. Wang, and T.N. Nguyen 2020, June · 2020
Cited alongside, same era.
Attention in Natural Language Processing
Galassi, A., M. Lippi, and P. Torroni. 2021, October · 2020
Cited alongside, same era.
Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges
Le, T.H.M., H. Chen, and M.A. Babar. 2020, June · 2020
Cited alongside, same era.
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., S.K. Deng, S. Fu, and N. Sundaresan 2020, November · 2020
Cited alongside, same era.
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., R. Krishna, Y. Ding, and B. Ray. 2022 · 2021
Cited alongside, same era.
Evaluating Large Language Models Trained on Code
Original
Chen, M., J. Tworek, H. Jun, Q. Yuan, H.P.d.O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F.P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W.H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A.N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. 2021, July · 2021
Cited alongside, same era.
GitHub Copilot · Your AI pair programmer
GitHub Inc. 2021 · 2021
Cited alongside, same era.
CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
Jiang, N., T. Lutellier, and L. Tan 2021, May · 2021
Cited alongside, same era.
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Original
Lu, S., D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S.K. Deng, S. Fu, and S. Liu. 2021, March · 2021
Cited alongside, same era.
Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code?
Asare, O., M. Nagappan, and N. Asokan. 2022 · 2022
Cited alongside, same era.
Grounded Copilot: How Programmers Interact with Code-Generating Models
Original
Barke, S., M.B. James, and N. Polikarpova. 2022, August · 2022
Cited alongside, same era.
To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?
Original
Ciniselli, M., L. Pascarella, and G. Bavota. 2022, April · 2022
Cited alongside, same era.

Then

Competition-Level Code Generation with AlphaCode
Li, Y., D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A.D. Lago, T. Hubert, P. Choy, C.d.M. d’Autume, I. Babuschkin, X. Chen, P.S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D.J. Mankowitz, E.S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals. 2022 · 2022
Closest in time.
An Empirical Evaluation of GitHub Copilot’s Code Suggestions
Nguyen, N. and S. Nadi 2022 · 2022
Closest in time.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. 2022 · 2022
Closest in time.
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri 2022, May · 2022
Closest in time.
Can OpenAI’s Codex Fix Bugs?: An evaluation on QuixBugs
Prenner, J., H. Babii, and R. Robbes 2022, May · 2022
Closest in time.
Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming
Sobania, D., M. Briesch, and F. Rothlauf 2022, July · 2022
Closest in time.
Open Source Security and Risk Analysis Report
Synopsys 2022 · 2022
Closest in time.
Code Faster with AI Completions
Tabnine. 2022 · 2022
Closest in time.
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
Vaithilingam, P., T. Zhang, and E.L. Glassman 2022, April · 2022
Closest in time.
A systematic evaluation of large language models of code
Xu, F.F., U. Alon, G. Neubig, and V.J. Hellendoorn 2022, June · 2022
Closest in time.
WhyGen: Explaining ML-powered Code Generation by Referring to Training Examples
Original
Yan, W. and Y. Li. 2022, April · 2022
Closest in time.
Repairing Bugs in Python Assignments Using Large Language Models
Zhang, J., J. Cambronero, S. Gulwani, V. Le, R. Piskac, G. Soares, and G. Verbruggen. 2022 · 2022
Closest in time.
Productivity assessment of neural code completion
Ziegler, A., E. Kalliamvakou, X.A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian 2022, June · 2022
Closest in time.
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H., B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt 2023, May · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…