2020

SCELMo: Source Code Embeddings from Language Models

Karampatsis, Rafael - Michael, Sutton, Charles

Understand

Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair.

Contextual embeddings are common in natural language processing but have not been previously applied in software engineering.
We introduce a new set of deep contextualized word representations for computer programs based on language models.
We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018).

Built on

Sequencer: Sequence-to-sequence learning for end-to-end program repair
Original
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus · 1901
Earlier work this paper cites.
A literature study of embeddings on source code
Original
Zimin Chen and Martin Monperrus · 1904
Earlier work this paper cites.
How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset
Original
Rafael-Michael Karampatsis and Charles Sutton · 1905
Earlier work this paper cites.
XLNet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le · 1906
Earlier work this paper cites.
Roberta: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 1907
Earlier work this paper cites.
Tricorder: Building a program analysis ecosystem
Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter · 1934
Earlier work this paper cites.
Word representations: A simple and general method for semi-supervised learning
Joseph Turian, Lev Ratinov, and Yoshua Bengio · 2010
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean · 2013
Earlier work this paper cites.
Defects4j: A database of existing faults to enable controlled testing studies for java programs
René Just, Darioush Jalali, and Michael D. Ernst · 2014
Earlier work this paper cites.
Efficient non-parametric estimation of multiple embeddings per word in vector space
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum · 2014
Earlier work this paper cites.

Similar

Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Cited alongside, same era.
Gated graph sequence neural networks
Yujia Li, Richard Zemel, Marc Brockschmidt, and Daniel Tarlow · 2016
Cited alongside, same era.
Automatic patch generation by learning correct code
Fan Long and Martin Rinard · 2016
Cited alongside, same era.
context2vec: Learning generic context embedding with bidirectional LSTM
Oren Melamud, Jacob Goldberger, and Ido Dagan · 2016
Cited alongside, same era.
Learning programs from noisy data
Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause · 2016
Cited alongside, same era.
Charagram: Embedding words and sentences via character n-grams
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu · 2016
Cited alongside, same era.
Enriching word vectors with subword information
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov · 2017
Cited alongside, same era.
Automatic software repair: A survey
L. Gazzola, D. Micucci, and L. Mariani · 2017
Cited alongside, same era.
Context2name: A deep learning-based approach to infer natural variable names from usage contexts
Original
Rohan Bavishi, Michael Pradel, and Koushik Sen · 2018
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton
Cited in the paper.
Learning to represent programs with graphs
Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi
Cited in the paper.

Then

How many of all bugs do we find? a study of static bug detectors
Andrew Habib and Michael Pradel · 2018
Later among the works it cites.
Automated software vulnerability detection with machine learning
Original
Jacob A. Harer, Louis Y. Kim, Rebecca L. Russell, Onur Ozdemir, Leonard R. Kosta, Akshay Rangamani, Lei H. Hamilton, Gabriel I. Centeno, Jonathan R. Key, Paul M. Ellingwood, Marc W. McConley, Jeffrey M. Opper, Sang Peter Chin, and Tomo Lazovich · 2018
Later among the works it cites.
Automatic software repair: A bibliography
Martin Monperrus · 2018
Later among the works it cites.
Deep contextualized word representations
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Later among the works it cites.
Deepbugs: A learning approach to name-based bug detection
Michael Pradel and Koushik Sen · 2018
Later among the works it cites.
Learning to represent edits
Original
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L. Gaunt · 2018
Later among the works it cites.
Code2vec: Learning distributed representations of code
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav · 2019
Later among the works it cites.
Neural program repair by jointly learning to localize and repair
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh singh · 2019
Later among the works it cites.
Sorting and transforming program repair ingredients via deep learning code similarities
Martin White, Michele Tufano, Matias Martinez, Martin Monperrus, and Denys Poshyvanyk · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…