2016

A Convolutional Attention Network for Extreme Summarization of Source Code

Allamanis, Miltiadis, Peng, Hao, Sutton, Charles

Understand

Attention mechanisms in neural networks have proved useful for problems in which the input and output do not have fixed dimension.

Often there exist features that are locally translation invariant and would be valuable for directing the model's attention, but previous attentional architectures are not constructed to learn such features specifically.
We introduce an attentional neural network that employs convolution on the input tokens to detect local time-invariant and long-range topical attention features in a context-dependent way.
We apply this architecture to the problem of extreme summarization of source code snippets into short, descriptive function name-like summaries.

Built on

The effects of comments and identifier names on program comprehensibility: an experimental investigation
Takang, Armstrong A, Grubb, Penny A, and Macredie, Robert D · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick · 1998
Earlier work this paper cites.
BLEU: a method for automatic evaluation of machine translation
Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu, Wei-Jing · 2002
Earlier work this paper cites.
Cognitive perspectives on the role of naming in computer programs
Liblit, Ben, Begel, Andrew, and Sweetser, Eve · 2006
Earlier work this paper cites.
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, Ronan and Weston, Jason · 2008
Earlier work this paper cites.
Rectified linear units improve restricted Boltzmann machines
Nair, Vinod and Hinton, Geoffrey E · 2010
Earlier work this paper cites.
GHTorrent: Github’s data from a firehose
Gousios, Georgios and Spinellis, Diomidis · 2012
Earlier work this paper cites.
On the naturalness of software
Hindle, Abram, Barr, Earl T, Su, Zhendong, Gabel, Mark, and Devanbu, Premkumar · 2012
Earlier work this paper cites.
Neural networks for machine learning
Hinton, Geoffrey, Srivastava, Nitsh, and Swersky, Kevin · 2012
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P · 2012
Earlier work this paper cites.
The impact of identifier style on effort and comprehension
Binkley, Dave, Davis, Marcia, Lawrie, Dawn, Maletic, Jonathan I, Morrell, Christopher, and Sharif, Bonita · 2013
Earlier work this paper cites.
Rectified linear units improve restricted Boltzmann machines
Maas, Andrew L., Hannun, Awni Y., and Ng, Andrew Y · 2013
Earlier work this paper cites.
Natural language models for predicting programming comments
Movshovitz-Attias, Dana, Cohen, WW, and W. Cohen, William · 2013
Earlier work this paper cites.
A statistical semantic language model for source code
Nguyen, Tung Thanh, Nguyen, Anh Tuan, Nguyen, Hoan Anh, and Nguyen, Tien N · 2013
Earlier work this paper cites.
On the importance of initialization and momentum in deep learning
Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey · 2013
Earlier work this paper cites.

Similar

Learning natural coding conventions
Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sutton, Charles · 2014
Cited alongside, same era.
A convolutional neural network for modelling sentences
Blunsom, Phil, Grefenstette, Edward, Kalchbrenner, Nal, et al · 2014
Cited alongside, same era.
On the properties of neural machine translation: Encoder–decoder approaches
Cho, Kyunghyun, van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua · 2014
Cited alongside, same era.
Modelling, visualising and summarising documents with a single convolutional neural network
Original
Denil, Misha, Demiraj, Alban, Kalchbrenner, Nal, Blunsom, Phil, and de Freitas, Nando · 2014
Cited alongside, same era.
Neural turing machines
Original
Graves, Alex, Wayne, Greg, and Danihelka, Ivo · 2014
Cited alongside, same era.
Phrase-based statistical translation of programming languages
Karaivanov, Svetoslav, Raychev, Veselin, and Vechev, Martin · 2014
Cited alongside, same era.
Structured generative models of natural source code
Maddison, Chris and Tarlow, Daniel · 2014
Cited alongside, same era.
Recurrent models of visual attention
Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al · 2014
Cited alongside, same era.
Migrating code with statistical machine translation
Nguyen, Anh Tuan, Nguyen, Tung Thanh, and Nguyen, Tien N · 2014
Cited alongside, same era.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, Nitish, Hinton, Geoffrey E., Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan · 2014
Cited alongside, same era.
On the localness of software
Tu, Zhaopeng, Su, Zhendong, and Devanbu, Premkumar · 2014
Cited alongside, same era.
Learning to execute
Original
Zaremba, Wojciech and Sutskever, Ilya · 2014
Cited alongside, same era.

Then

Transition-based dependency parsing with stack long short-term memory
Dyer, Chris, Ballesteros, Miguel, Ling, Wang, Matthews, Austin, and Smith, Noah A · 2015
Later among the works it cites.
Learning to transduce with unbounded memory
Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil · 2015
Later among the works it cites.
Delving deep into rectifiers: surpassing human-level performance on ImageNet classification
Original
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian · 2015
Later among the works it cites.
Teaching machines to read and comprehend
Hermann, Karl Moritz, Kocisky, Tomas, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil · 2015
Later among the works it cites.
Inferring algorithmic patterns with stack-augmented recurrent nets
Original
Joulin, Armand and Mikolov, Tomas · 2015
Later among the works it cites.
Neural programmer: Inducing latent programs with gradient descent
Original
Neelakantan, Arvind, Le, Quoc V, and Sutskever, Ilya · 2015
Later among the works it cites.
Learning to generate pseudo-code from source code using statistical machine translation
Oda, Yusuke, Fudaba, Hiroyuki, Neubig, Graham, Hata, Hideaki, Sakti, Sakriani, Toda, Tomoki, and Nakamura, Satoshi · 2015
Later among the works it cites.
Learning program embeddings to propagate feedback on student code
Piech, Chris, Huang, Jonathan, Nguyen, Andy, Phulsuksombati, Mike, Sahami, Mehran, and Guibas, Leonidas J · 2015
Later among the works it cites.
Predicting program properties from “big code”
Raychev, Veselin, Vechev, Martin, and Krause, Andreas · 2015
Later among the works it cites.
Neural programmer-interpreters
Original
Reed, Scott and de Freitas, Nando · 2015
Later among the works it cites.
A neural attention model for abstractive sentence summarization
Rush, Alexander M, Chopra, Sumit, and Weston, Jason · 2015
Later among the works it cites.
Going deeper with convolutions
Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew · 2015
Later among the works it cites.
Pointer networks
Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep · 2015
Later among the works it cites.
Show, attend and tell: Neural image caption generation with visual attention
Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun, Courville, Aaron C, Salakhutdinov, Ruslan, Zemel, Richard S, and Bengio, Yoshua · 2015
Later among the works it cites.
Convolutional neural networks over tree structures for programming language processing
Mou, Lili, Li, Ge, Zhang, Lu, Wang, Tao, and Jin, Zhi · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…