2023

AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation

Murali, Vijayaraghavan, Maddila, Chandra, Ahmad, Imad et al.

Understand

Generative LLMs have been shown to effectively power AI-based code authoring tools that can suggest entire statements or blocks of code during code authoring.

In this paper we present CodeCompose, an AI-assisted code authoring tool developed and deployed at Meta internally.
CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality.
We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 9 programming languages and several coding surfaces.

Built on

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020 · 1909
Earlier work this paper cites.
BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL ’02) . Association for Computational Linguistics, USA, 311–318
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Language Models are Few-Shot Learners
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2005
Earlier work this paper cites.
Thrift: Scalable cross-language services implementation
Mark Slee, Aditya Agarwal, and Marc Kwiatkowski. 2007 · 2007
Earlier work this paper cites.
How Program History Can Improve Code Completion. In 2008 23rd IEEE/ACM International Conference on Automated Software Engineering . 317–326
R. Robles and M. Lanza. 2008 · 2008
Earlier work this paper cites.
Learning from examples to improve code completion systems. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering (ESEC/FSE ’09)
Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009 · 2009
Earlier work this paper cites.
Technical Action Research as a Validation Method in Information Systems Design Science. In Design Science Research in Information Systems. Advances in Theory and Practice , Ken Peffers, Marcus Rothenberger, and Bill Kuechler (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 220–238
Roel Wieringa and Ayşe Moralı. 2012 · 2012
Earlier work this paper cites.
Flow Javascript
Accessed 2023 · 2014
Earlier work this paper cites.
An Eye-Tracking Study of User Interactions with Query Auto Completion (CIKM ’14) . Association for Computing Machinery, New York, NY, USA, 549–558
Kajta Hofmann, Bhaskar Mitra, Filip Radlinski, and Milad Shokouhi. 2014 · 2014
Earlier work this paper cites.
Intelligent Code Completion with Bayesian Networks
Sebastian Proksch, Johannes Lerch, and Mira Mezini. 2015 · 2015
Earlier work this paper cites.
Designing and conducting mixed methods research
John W Creswell and Vicki L Plano Clark. 2017 · 2017
Earlier work this paper cites.
Attention Is All You Need
Original
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.

Similar

When Code Completion Fails: A Case Study on Real-World Completions. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . 960–970
Vincent J. Hellendoorn, Sebastian Proksch, Harald C. Gall, and Alberto Bacchelli. 2019 · 2019
Cited alongside, same era.
Github Copilot
Github Accessed 2021 · 2021
Cited alongside, same era.
Microsoft Intellicode
Microsoft Accessed 2021 · 2021
Cited alongside, same era.
Program Synthesis with Large Language Models
Original
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021 · 2021
Cited alongside, same era.
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021
Cited alongside, same era.
Code Prediction by Feeding Trees to Transformers. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . 150–162
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021 · 2021
Cited alongside, same era.
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021 · 2021
Cited alongside, same era.
How GitHub Copilot helps improve developer productivity
Github 2022 · 2022
Cited alongside, same era.
ML Enhanced Code Completion
Google Accessed 2021 · 2022
Cited alongside, same era.
Efficient Training of Language Models to Fill in the Middle
Original
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022 · 2022
Cited alongside, same era.
An empirical evaluation of GitHub copilot’s code suggestions. In Proceedings of the 19th International Conference on Mining Software Repositories (MSR ’22)
Nhan Nguyen and Sarah Nadi. 2022 · 2022
Cited alongside, same era.
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22)
Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022 · 2022
Cited alongside, same era.

Then

Productivity Assessment of Neural Code Completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (San Diego, CA, USA) (MAPS 2022) . Association for Computing Machinery, New York, NY, USA, 21–29
Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022 · 2022
Later among the works it cites.
GitHub CoPilot’s economic impact
Accessed 2021 · 2023
Closest in time.
Amazon CodeWhisperer
Amazon Accessed 2023 · 2023
Closest in time.
Language Server Protocol (LSP)
Microsoft Accessed 2023 · 2023
Closest in time.
Grounded Copilot: How Programmers Interact with Code-Generating Models
Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023 · 2023
Closest in time.
InCoder: A Generative Model for Code Infilling and Synthesis
Original
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023 · 2023
Closest in time.
Hallucinations in Large Multilingual Translation Models
Original
Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023 · 2023
Closest in time.
Survey of Hallucination in Natural Language Generation
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023 · 2023
Closest in time.
The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucination
Original
Zihao Li. 2023 · 2023
Closest in time.
Improving ChatGPT Prompt for Code Generation
Original
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. 2023 · 2023
Closest in time.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Original
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023 · 2023
Closest in time.
Code Llama: Open Foundation Models for Code
Original
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023 · 2023
Closest in time.
Multi-line AI-assisted Code Authoring
Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Maddila, Vijayaraghavan Murali, Ali Tayyebi, and Nachiappan Nagappan. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…