Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009

Earlier work this paper cites.

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021

Earlier work this paper cites.

Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 4582–4597

Xiang Lisa Li and Percy Liang. 2021 · 2021

Earlier work this paper cites.

Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions

Original

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021 · 2021

Earlier work this paper cites.

Introducing ChatGPT

Open AI. 2022 · 2022

Earlier work this paper cites.

An Empirical Study of Code Smells in Transformer-based Code Generation Techniques. 71–82

Mohammed Latif Siddiq, Shafayat Majumder, Maisha Mim, Sourov Jajodia, and Joanna Cecilia da Silva Santos. 2022 · 2022

Earlier work this paper cites.

SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security (Singapore, Singapore) (MSR4P&S 2022) . Association for Computing Machinery, New York, NY, USA, 29–33

Mohammed Latif Siddiq and Joanna C. S. Santos. 2022 · 2022

Earlier work this paper cites.

Assessing the quality of GitHub copilot’s code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (Singapore, Singapore) (PROMISE 2022) . Association for Computing Machinery, New York, NY, USA, 62–71

Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022 · 2022

Earlier work this paper cites.

Multi-lingual Evaluation of Code Generation Models

Original

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023 · 2023

Earlier work this paper cites.

Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (<conf-loc>, <city>Toronto ON</city>, <country>Canada</country>, </conf-loc>) (SIGCSE 2023) . Association for Computing Machinery, New York, NY, USA, 1136–1142

Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023 · 2023

Cited alongside, same era.

Prompt Engineering with ChatGPT: A Guide for Academic Writers

L. Giray. 2023 · 2023

Cited alongside, same era.

Large Language Models for Code: Security Hardening and Adversarial Testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23) . ACM

Jingxuan He and Martin Vechev. 2023 · 2023

Cited alongside, same era.

A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges

Original

Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2023 · 2023

Cited alongside, same era.

LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . 588–592

Catherine Tony, Markus Mutas, Nicolás E. Díaz Ferreyra, and Riccardo Scandariato. 2023 · 2023

Later among the works it cites.

Small language models improve giants by rewriting their outputs

Original

Giorgos Vernikos, Arthur Bražinskas, Jakub Adamek, Jonathan Mallinson, Aliaksei Severyn, and Eric Malmi. 2023 · 2023

Later among the works it cites.

Factcheck-GPT: End-to-End Fine-Grained Document-Level Fact-Checking and Correction of LLM Output

Original

Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2023 · 2023

Later among the works it cites.

A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT

Original

Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023 · 2023