2020

IntelliCode Compose: Code Generation Using Transformer

Svyatkovskiy, Alexey, Deng, Shao Kun, Fu, Shengyu et al.

Understand

In software development through integrated development environments (IDEs), code completion is one of the most widely used features.

Nevertheless, majority of integrated development environments only support completion of methods and APIs, or arguments.
In this paper, we introduce IntelliCode Compose $-$ a general-purpose multilingual code completion tool which is capable of predicting sequences of code tokens of arbitrary types, generating up to entire lines of syntactically correct code.
It leverages state-of-the-art generative transformer model trained on 1.2 billion lines of source code in Python, $C\#$, JavaScript and TypeScript programming languages.

Built on

ROUGE: A package for automatic evaluation of summaries
Chin-Yew Lin · 2004
Earlier work this paper cites.
Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics
Chin-Yew Lin and Franz Josef Och · 2004
Earlier work this paper cites.
Learning from examples to improve code completion systems
Marcel Bruch, Martin Monperrus, and Mira Mezini · 2009
Earlier work this paper cites.
On the naturalness of software
Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu · 2012
Earlier work this paper cites.
Automatic parameter recommendation for practical api usage
Cheng Zhang, Juyuan Yang, Yi Zhang, Jing Fan, Xin Zhang, Jianjun Zhao, and Peizhao Ou · 2012
Earlier work this paper cites.
A statistical semantic language model for source code
Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N Nguyen · 2013
Earlier work this paper cites.
Cscc: Simple, efficient, context sensitive code completion
Muhammad Asaduzzaman, Chanchal Roy, Kevin Schneider, and Daqing Hou · 2014
Earlier work this paper cites.
Code completion with statistical language models
Veselin Raychev, Martin Vechev, and Eran Yahav · 2014
Earlier work this paper cites.
Distilling the knowledge in a neural network, 2015
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Intelligent code completion with bayesian networks
Sebastian Proksch, Johannes Lerch, and Mira Mezini · 2015
Earlier work this paper cites.
Deep API learning
Original
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim · 2016
Earlier work this paper cites.
Automatically learning semantic features for defect prediction
Song Wang, Taiyue Liu, and Lin Tan · 2016
Earlier work this paper cites.
Nomen est omen: Exploring and exploiting similarities between argument and parameter names
Hui Liu, Qiurong Liu, Cristian-Alexandru Staicu, Michael Pradel, and Yue Luo · 2016
Earlier work this paper cites.
Layer normalization, 2016
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton · 2016
Earlier work this paper cites.
Tying word vectors and word classifiers: A loss framework for language modeling
Original
Hakan Inan, Khashayar Khosravi, and Richard Socher · 2016
Earlier work this paper cites.

Similar

A corpus and cloze evaluation for deeper understanding of commonsense stories
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen · 2016
Cited alongside, same era.
Attention is all you need, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
A survey of machine learning for big code and naturalness
Original
Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles A. Sutton · 2017
Cited alongside, same era.
Code Recommenders
Eclipse Foundation · 2017
Cited alongside, same era.
Eclipse SnipMatch
Eclipse Recommenders · 2017
Cited alongside, same era.
Deep code search
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim · 2018
Cited alongside, same era.
code2seq: Generating sequences from structured representations of code
Original
Uri Alon, Omer Levy, and Eran Yahav · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford · 2018
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Horovod: fast and easy distributed deep learning in TensorFlow
Original
Alexander Sergeev and Mike Del Balso · 2018
Cited alongside, same era.
Swag: A large-scale adversarial dataset for grounded commonsense inference, 2018
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi · 2018
Cited alongside, same era.
Program synthesis and semantic parsing with learned code idioms
Original
Richard Shin, Miltiadis Allamanis, Marc Brockschmidt, and Oleksandr Polozov · 2019
Cited alongside, same era.

Then

Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le · 2019
Later among the works it cites.
Roberta: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Later among the works it cites.
The evolved transformer
Original
David R. So, Chen Liang, and Quoc V. Le · 2019
Later among the works it cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2019
Later among the works it cites.
Autocompletion with deep learning
Jacob Jackson · 2019
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2019
Later among the works it cites.
Ctrl: A conditional transformer language model for controllable generation, 2019
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher · 2019
Later among the works it cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew · 2019
Later among the works it cites.
Openwebtext corpus
Aaron Gokaslan and Vanya Cohen · 2019
Later among the works it cites.
Distilling task-specific knowledge from bert into simple neural networks, 2019
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin · 2019
Later among the works it cites.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf · 2019
Later among the works it cites.
Fast and memory-efficient neural code completion, 2020
Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Franco, and Miltiadis Allamanis · 2020
Closest in time.
Ai-assisted development
Microsoft Corporation · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…