2022

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants

Sandoval, Gustavo, Pearce, Hammond, Nys, Teo et al.

Understand

Large Language Models (LLMs) such as OpenAI Codex are increasingly being used as AI-based coding assistants.

Understanding the impact of these tools on developers' code is paramount, especially as recent work showed that LLMs may suggest cybersecurity vulnerabilities.
We conduct a security-driven user study (N=58) to assess code written by student programmers when assisted by LLMs.
Given the potential severity of low-level bugs as well as their relative frequency in real-world projects, we tasked participants with implementing a singly-linked 'shopping list' structure in C.

Built on

S. Imai, “Is GitHub Copilot a Substitute for Human Pair-programming? An Empirical Study,” in 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) , May 2022, pp. 319–321, iSSN: 2574-1926
1926
Earlier work this paper cites.
P. Gage, “A New Algorithm for Data Compression,” C Users Journal , vol. 12, no. 2, pp. 23–38, Feb. 1994, place: USA Publisher: R & D Publications, Inc
1994
Earlier work this paper cites.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” Sep. 2020, arXiv:2002.08155 [cs]. [Online]. Available: http://arxiv.org/abs/2002.08155
Original
2002
Earlier work this paper cites.
H. Fangohr, “A Comparison of C, MATLAB, and Python as Teaching Languages in Engineering,” in Computational Science - ICCS 2004 , ser. Lecture Notes in Computer Science, M. Bubak, G. D. van Albada, P. M. A. Sloot, and J. Dongarra, Eds. Berlin, Heidelberg: Springer, 2004, pp. 1210–1217
2004
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv:2005.14165 [cs] , Jul. 2020, arXiv: 2005.14165. [Online]. Available: http://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
H. M. J. Hung, S.-J. Wang, and R. O’Neill, “A Regulatory Perspective on Choice of Margin and Statistical Inference Issue in Non-inferiority Trials,” Biometrical Journal , vol. 47, no. 1, pp. 28–36, 2005. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/bimj.200410084
2005
Earlier work this paper cites.
M. Pistoia, S. Chandra, S. J. Fink, and E. Yahav, “A survey of static analysis methods for identifying security vulnerabilities in software systems,” IBM Systems Journal , vol. 46, no. 2, pp. 265–288, 2007, conference Name: IBM Systems Journal
2007
Earlier work this paper cites.
E. Walker and A. S. Nowacki, “Understanding Equivalence and Noninferiority Testing,” Journal of General Internal Medicine , vol. 26, no. 2, pp. 192–196, Feb. 2011. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3019319/
2011
Earlier work this paper cites.
K. Serebryany, D. Bruening, A. Potapenko, and D. Vyukov, “AddressSanitizer: A Fast Address Sanity Checker,” in 2012 USENIX Annual Technical Conference (USENIX ATC 12) . Boston, MA: USENIX Association, Jun. 2012, pp. 309–318. [Online]. Available: https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany
2012
Earlier work this paper cites.
M. Polacek, “GCC Undefined Behavior Sanitizer - ubsan,” Oct. 2014. [Online]. Available: https://developers.redhat.com/blog/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan
2014
Earlier work this paper cites.
A. J. Ko, T. D. LaToza, and M. M. Burnett, “A practical guide to controlled experiments of software engineering tools with human participants,” Empirical Software Engineering , vol. 20, no. 1, pp. 110–141, Feb. 2015. [Online]. Available: https://doi.org/10.1007/s10664-013-9279-3
2015
Earlier work this paper cites.
I. Salman, A. T. Misirli, and N. Juristo, “Are Students Representatives of Professionals in Software Engineering Experiments?” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1, May 2015, pp. 666–676, iSSN: 1558-1225
2015
Earlier work this paper cites.
A. Assaraf, “This is what your developers are doing 75% of the time, and this is the cost you pay,” Feb. 2015. [Online]. Available: https://coralogix.com/blog/this-is-what-your-developers-are-doing-75-of-the-time-and-this-is-the-cost-you-pay/
2015
Earlier work this paper cites.
M. Aizatsky, K. Serebryany, O. Chang, A. Arya, and M. Whittaker, “Announcing OSS-Fuzz: Continuous fuzzing for open source software,” Dec. 2016. [Online]. Available: https://opensource.googleblog.com/2016/12/announcing-oss-fuzz-continuous-fuzzing.html
2016
Earlier work this paper cites.
M. di Biase, M. Bruntink, and A. Bacchelli, “A Security Perspective on Code Review: The Case of Chromium,” in 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM) , Oct. 2016, pp. 21–30, iSSN: 2470-6892
2016
Earlier work this paper cites.
Y. Acar, C. Stransky, D. Wermke, M. L. Mazurek, and S. Fahl, “Security Developer Studies with GitHub Users: Exploring a Convenience Sample,” 2017, pp. 81–95. [Online]. Available: https://www.usenix.org/conference/soups2017/technical-sessions/presentation/acar
2017
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” p. 24, 2019. [Online]. Available: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
2019
Earlier work this paper cites.

Similar

G. Thomas, “A proactive approach to more secure code – Microsoft Security Response Center,” Jul. 2019. [Online]. Available: https://msrc-blog.microsoft.com/2019/07/16/a-proactive-approach-to-more-secure-code/
2019
Cited alongside, same era.
M. Zalewski, “American fuzzy lop,” Jul. 2020. [Online]. Available: https://github.com/google/AFL
2020
Cited alongside, same era.
K. Tuma, L. Sion, R. Scandariato, and K. Yskout, “Automating the early detection of security design flaws,” in Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems , ser. MODELS ’20. New York, NY, USA: Association for Computing Machinery, Oct. 2020, pp. 332–342. [Online]. Available: https://doi.org/10.1145/3365438.3410954
2020
Cited alongside, same era.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating Large Language Models Trained on Code,” arXiv:2107.03374 [cs] , Jul. 2021, arXiv: 2107.03374. [Online]. Available: http://arxiv.org/abs/2107.03374
Original
2021
Cited alongside, same era.
O. Lieber, O. Sharir, B. Lentz, and Y. Shoham, “Jurassic-1: Technical Details and Evaluation,” AI21 Labs, Tech. Rep., Aug. 2021. [Online]. Available: https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
2021
Cited alongside, same era.
AI21, “Jurassic-1 Language Models - AI21 Studio Docs,” 2021. [Online]. Available: https://studio.ai21.com/docs/jurassic1-language-models/#general-purpose-models
2021
Cited alongside, same era.
G. Inc., “CodeQL for research,” 2021. [Online]. Available: https://securitylab.github.com/tools/codeql/
2021
Cited alongside, same era.
A. Aiken, “A System for Detecting Software Similarity,” Apr. 2021. [Online]. Available: https://theory.stanford.edu/~aiken/moss/
2021
Cited alongside, same era.
J. P. Wahle, T. Ruas, N. Meuschke, and B. Gipp, “Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection,” in 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) , Sep. 2021, pp. 226–229
2021
Cited alongside, same era.
P. E. Black, B. Guttman, and V. Okun, “Guidelines on Minimum Standards for Developer Verification of Software,” National Institute of Standards and Technology, Tech. Rep., Oct. 2021. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8397.pdf
2021
Cited alongside, same era.
V. J. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo, “The Art, Science, and Engineering of Fuzzing: A Survey,” IEEE Transactions on Software Engineering , vol. 47, no. 11, pp. 2312–2331, Nov. 2021, conference Name: IEEE Transactions on Software Engineering
2021
Cited alongside, same era.
R. Torres, “GitHub Copilot adds 400K subscribers in first month,” Aug. 2022. [Online]. Available: https://www.ciodive.com/news/github-copilot-microsoft-software-developer/628587/
2022
Cited alongside, same era.

Then

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A Conversational Paradigm for Program Synthesis,” Mar. 2022, arXiv:2203.13474 [cs]. [Online]. Available: http://arxiv.org/abs/2203.13474
Original
2022
Closest in time.
J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather, “The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming,” in Australasian Computing Education Conference . Virtual Event Australia: ACM, Feb. 2022, pp. 10–19. [Online]. Available: https://dl.acm.org/doi/10.1145/3511861.3511863
2022
Closest in time.
S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models,” in Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 , ser. ICER ’22. New York, NY, USA: Association for Computing Machinery, Aug. 2022, pp. 27–43. [Online]. Available: https://doi.org/10.1145/3501385.3543957
2022
Closest in time.
A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, Z. Ming, and Jiang, “GitHub Copilot AI pair programmer: Asset or Liability?” Jun. 2022, arXiv:2206.15331 [cs]. [Online]. Available: http://arxiv.org/abs/2206.15331
Original
2022
Closest in time.
P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models,” in CHI Conference on Human Factors in Computing Systems Extended Abstracts . New Orleans LA USA: ACM, Apr. 2022, pp. 1–7. [Online]. Available: https://dl.acm.org/doi/10.1145/3491101.3519665
2022
Closest in time.
A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , ser. MAPS 2022. New York, NY, USA: Association for Computing Machinery, Jun. 2022, pp. 21–29. [Online]. Available: https://doi.org/10.1145/3520312.3534864
2022
Closest in time.
M. Tabachnyk and S. Nikolov, “ML-Enhanced Code Completion Improves Developer Productivity,” Jul. 2022. [Online]. Available: http://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html
2022
Closest in time.
B. Developers, “Welcome to Bandit — Bandit documentation,” 2022. [Online]. Available: https://bandit.readthedocs.io/en/latest/
2022
Closest in time.
N. A. Ernst and G. Bavota, “AI-Driven Development Is Here: Should You Worry?” IEEE Software , vol. 39, no. 2, pp. 106–110, Mar. 2022, conference Name: IEEE Software
2022
Closest in time.
M. Ciniselli, L. Pascarella, and G. Bavota, “To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?” Apr. 2022, arXiv:2204.06894 [cs]. [Online]. Available: http://arxiv.org/abs/2204.06894
Original
2022
Closest in time.
C. Topham, “Publication of the FSF-funded white papers on questions around Copilot,” Feb. 2022. [Online]. Available: https://www.fsf.org/news/publication-of-the-fsf-funded-white-papers-on-questions-around-copilot
2022
Closest in time.
S. Biderman and E. Raff, “Neural Language Models are Effective Plagiarists,” Jan. 2022, arXiv:2201.07406 [cs]. [Online]. Available: http://arxiv.org/abs/2201.07406
Original
2022
Closest in time.
M. Tahaei and K. Vaniea, “Recruiting Participants With Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing List,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , ser. CHI ’22. New York, NY, USA: Association for Computing Machinery, Apr. 2022, pp. 1–15. [Online]. Available: https://doi.org/10.1145/3491102.3501957
2022
Closest in time.
J.-B. Döderlein, M. Acher, D. E. Khelladi, and B. Combemale, “Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic?” Oct. 2022, arXiv:2210.14699 [cs]. [Online]. Available: http://arxiv.org/abs/2210.14699
Original
2022
Closest in time.
M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen, “Efficient Training of Language Models to Fill in the Middle,” Jul. 2022, arXiv:2207.14255 [cs]. [Online]. Available: http://arxiv.org/abs/2207.14255
Original
2022
Closest in time.
H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining Zero-Shot Vulnerability Repair with Large Language Models,” in 2023 IEEE Symposium on Security and Privacy (SP) . Los Alamitos, CA, USA: IEEE Computer Society, May 2023, pp. 1–18. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/SP46215.2023.00001
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…