2022

Fix Bugs with Transformer through a Neural-Symbolic Edit Grammar

Hu, Yaojie, Shi, Xingjian, Zhou, Qiang et al.

Understand

We introduce NSEdit (neural-symbolic edit), a novel Transformer-based code repair method.

Given only the source code that contains bugs, NSEdit predicts an editing sequence that can fix the bugs.
The edit grammar is formulated as a regular language, and the Transformer uses it as a neural-symbolic scripting interface to generate editing programs.
We modify the Transformer and add a pointer network to select the edit locations.

Built on

Neural program repair by jointly learning to localize and repair
Original
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh · 1904
Earlier work this paper cites.
Speech understanding systems: A summary of results of the five-year research effort
D Raj Reddy et al · 1977
Earlier work this paper cites.
Pattern-matching-the gestalt approach
John W Ratcliff and David E Metzener · 1988
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
Ronald J Williams and David Zipser · 1989
Earlier work this paper cites.
Bagging predictors
Leo Breiman · 1996
Earlier work this paper cites.
Combining predictions for accurate recommender systems
Michael Jahrer, Andreas Töscher, and Robert Legenstein · 2010
Earlier work this paper cites.
Sequence transduction with recurrent neural networks
Original
Alex Graves · 2012
Earlier work this paper cites.
Mining Source Code Repositories at Massive Scale using Language Modeling
Miltiadis Allamanis and Charles Sutton · 2013
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Earlier work this paper cites.
Siamese neural networks for one-shot image recognition
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al · 2015
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015
Earlier work this paper cites.
Pointer networks
Original
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly · 2015
Earlier work this paper cites.
Professor forcing: A new algorithm for training recurrent networks
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio · 2016
Earlier work this paper cites.
Probabilistic model for code with decision trees
Veselin Raychev, Pavol Bielik, and Martin Vechev · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Neuro-symbolic program corrector for introductory programming assignments
Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh · 2018
Earlier work this paper cites.

Similar

Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
A retrieve-and-edit framework for predicting structured outputs
Original
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang · 2018
Cited alongside, same era.
Tune: A research platform for distributed model selection and training
Original
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever · 2018
Cited alongside, same era.
Train on validation: squeezing the data lemon
Original
Guy Tennenholtz, Tom Zahavy, and Shie Mannor · 2018
Cited alongside, same era.
Learning to represent edits
Original
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt · 2018
Cited alongside, same era.
Sequencer: Sequence-to-sequence learning for end-to-end program repair
Zimin Chen, Steve James Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus · 2019
Cited alongside, same era.
Getafix: learning to fix bugs automatically
Bader Johannes, Scott Andrew, Pradel Michael, and Chandra Satish · 2019
Cited alongside, same era.
Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement
Wouter Kool, Herke Van Hoof, and Max Welling · 2019
Cited alongside, same era.
The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
Original
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu · 2019
Cited alongside, same era.
Facebook fair’s wmt19 news translation task submission
Original
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov · 2019
Cited alongside, same era.
An empirical study on learning bug-fixing patches in the wild via neural machine translation
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk · 2019
Cited alongside, same era.

Then

Learning and evaluating contextual embedding of source code
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi · 2020
Later among the works it cites.
Emergent linguistic structure in artificial neural networks trained by self-supervision
Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy · 2020
Later among the works it cites.
Learning to fix build errors with graph2diff neural networks
Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian · 2020
Later among the works it cites.
Are transformers universal approximators of sequence-to-sequence functions?
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar · 2020
Later among the works it cites.
Unified pre-training for program understanding and generation
Original
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang · 2021
Later among the works it cites.
Self-supervised bug detection and repair
Original
Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt · 2021
Later among the works it cites.
Tfix: Learning to fix coding errors with a text-to-text transformer
Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev · 2021
Later among the works it cites.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al · 2021
Later among the works it cites.
Deepdebug: Fixing python bugs using stack traces, backtranslation, and code skeletons
Original
Dawn Drain, Colin B Clement, Guillermo Serrato, and Neel Sundaresan · 2021
Later among the works it cites.
Discriminative reranking for neural machine translation
Ann Lee, Michael Auli, and Marc’Aurelio Ranzato · 2021
Later among the works it cites.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al · 2021
Later among the works it cites.
Cotext: Multi-task learning with code-text transformer
Original
Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Anibal, Alec Peltekian, and Yanfang Ye · 2021
Later among the works it cites.
On the generalizability of neural program models with respect to semantic-preserving program transformations
Md Rafiqul Islam Rabin, Nghi DQ Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour · 2021
Later among the works it cites.
Zero-shot text-to-image generation
Original
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever · 2021
Later among the works it cites.
Msa transformer
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives · 2021
Later among the works it cites.
Learning structural edits via incremental tree transformations
Original
Ziyu Yao, Frank F Xu, Pengcheng Yin, Huan Sun, and Graham Neubig · 2021
Later among the works it cites.
Break-it-fix-it: Unsupervised learning for program repair
Original
Michihiro Yasunaga and Percy Liang · 2021
Later among the works it cites.
A syntax-guided edit decoder for neural program repair
Original
Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…