Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study

Language models are few-shot learners. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems (NeurIPS) . 1877–1901

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

A Coefficient of Agreement for Nominal Scales

Jacob Cohen. 1960 · 1960

Earlier work this paper cites.

PROW: A step toward automatic program writing. In Proceedings of the 1st International Joint Conference on Artificial Intelligence (IJCAI) . ACM, 241–252

Richard J Waldinger and Richard CT Lee. 1969 · 1969

Earlier work this paper cites.

Static and dynamic analysis: Synergy and duality. In Proceedings of the ICSE Workshop on Dynamic Analysis (WODA) . ACM, 24–27

Michael D Ernst. 2003 · 2003

Earlier work this paper cites.

Selecting Empirical Methods for Software Engineering Research

Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. 2008 · 2008

Earlier work this paper cites.

Guidelines for conducting and reporting case study research in software engineering

Per Runeson and Martin Höst. 2009 · 2009

Earlier work this paper cites.

A systematic mapping study on dynamic metrics and software quality. In 2012 28th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 326–335

Amjed Tahir and Stephen G MacDonell. 2012 · 2012

Earlier work this paper cites.

Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement

John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013 · 2013

Earlier work this paper cites.

A comparative analysis of node. js (server-side javascript)

Nimesh Chhetri. 2016 · 2016

Earlier work this paper cites.

A systematic mapping study of software development with GitHub

Valerio Cosentino, Javier L Cánovas Izquierdo, and Jordi Cabot. 2017 · 2017

Earlier work this paper cites.

Comment on: “Comparison of periodized and non-periodized resistance training on maximal strength: A meta-analysis”

Joao Pedro Nunes, Alex S Ribeiro, Brad J Schoenfeld, and Edilson S Cyrino. 2018 · 2018

Earlier work this paper cites.

Lessons from building static analysis tools at google

Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018 · 2018

Earlier work this paper cites.

The adoption of javascript linters in practice: A case study on eslint

Kristín Fjóla Tómasdóttir, Mauricio Aniche, and Arie Van Deursen. 2018 · 2018

Earlier work this paper cites.

Scaling static analyses at Facebook

Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W O’Hearn. 2019 · 2019

Earlier work this paper cites.

Static analysis for proactive security

Michael Huth and Flemming Nielson. 2019 · 2019

Earlier work this paper cites.

Python for data analytics, scientific and technical applications. In Proceedings of the Amity international conference on artificial intelligence (AICAI) . IEEE, 140–145

Abhinav Nagpal and Goldie Gabrani. 2019 · 2019

Earlier work this paper cites.

Why do software developers use static analysis tools? a user-centered study of developer needs and motivations

Lisa Nguyen Quang Do, James R Wright, and Karim Ali. 2020 · 2020

Earlier work this paper cites.

A comparative study of static code analysis tools for vulnerability detection in c/c++ and java source code

Arvinder Kaur and Ruchikaa Nayyar. 2020 · 2020

Earlier work this paper cites.

SourceFinder: Finding Malware Source-Code from Publicly Available Repositories in GitHub. In Proceedings of the 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID) . USENIX, 149–163

Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelos E Papalexakis, and Michalis Faloutsos. 2020 · 2020

Earlier work this paper cites.

On the recall of static call graph construction in practice. In Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE) . ACM, 1049–1060

Li Sui, Jens Dietrich, Amjed Tahir, and George Fourtounis. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021

Earlier work this paper cites.

GitHub’s Copilot Security Issues

Alex Dow. 2021 · 2021

Earlier work this paper cites.

GitHub Copilot: Security and Privacy

Dwayne McDaniel. 2021 · 2021

Earlier work this paper cites.

Introducing Codex: The AI Behind GitHub Copilot

OpenAI. 2021 · 2021

Earlier work this paper cites.

The secret life of software vulnerabilities: A large-scale empirical study

Emanuele Iannone, Roberta Guadagni, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2022 · 2022

Earlier work this paper cites.

Detecting false alarms from automatic static analysis tools: How far are we?. In Proceedings of the 44th International Conference on Software Engineering (ICSE) . ACM, 698–709

Hong Jin Kang, Khai Loong Aw, and David Lo. 2022 · 2022

Earlier work this paper cites.

An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode. In Proceedings of the 16th IEEE International Workshop on Software Clones (IWSC) . IEEE, 10–15

Sila Lertbanjongngam, Bodin Chinthanet, Takashi Ishio, Raula Gaikovina Kula, Pattara Leelaprute, Bundit Manaskasemsak, Arnon Rungsawang, and Kenichi Matsumoto. 2022 · 2022

Earlier work this paper cites.

Competition-level code generation with alphacode

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al · 2022

Earlier work this paper cites.

An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th IEEE/ACM International Conference on Mining Software Repositories (MSR) . IEEE, 1–5

Nhan Nguyen and Sarah Nadi. 2022 · 2022

Cited alongside, same era.

Asleep at the keyboard? assessing the security of github copilot’s code contributions. In Proceedings of the 43rd IEEE Symposium on Security and Privacy (SP) . IEEE, 754–768

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022 · 2022

Cited alongside, same era.

Python vs. JavaScript: Which Should You Learn?

Meghan Reichenbach. 2022 · 2022

Cited alongside, same era.

Security Implications of Large Language Model Code Assistants: A User Study

Original

Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2022 · 2022

Cited alongside, same era.

What is it like to program with artificial intelligence?

Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey

Original

Xinyu She, Yue Liu, Yanjie Zhao, Yiling He, Li Li, Chakkrit Tantithamthavorn, Zhan Qin, and Haoyu Wang. 2023 · 2023

Closest in time.

CWE VIEW: Software Development

The MITRE Corporation. 2023 · 2023

Closest in time.

Llmseceval: A dataset of natural language prompts for security evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 588–592

Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato. 2023 · 2023

Closest in time.

A prompt pattern catalog to enhance prompt engineering with chatgpt

Original

Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023 · 2023

Closest in time.

Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT