2022

Compilable Neural Code Generation with Compiler Feedback

Wang, Xin, Wang, Yasheng, Wan, Yao et al.

Understand

Automatically generating compilable programs with (or without) natural language descriptions has always been a touchstone problem for computational linguistics and automated software engineering.

Existing deep-learning approaches model code generation as text generation, either constrained by grammar structures in decoder, or driven by pre-trained language models on large-scale code corpus (e.g., CodeGPT, PLBART, and CodeT5).
However, few of them account for compilability of the generated programs.
To improve compilability of the generated programs, this paper proposes COMPCODER, a three-stage pipeline utilizing compiler feedback for compilable code generation, including language model fine-tuning, compilability reinforcement, and compilability discrimination.

Built on

Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 1909
Earlier work this paper cites.
Fine-tuning language models from human preferences
Original
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019 · 1909
Earlier work this paper cites.
Sequencer: Sequence-to-sequence learning for end-to-end program repair
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2021 · 1959
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J. Williams. 1992 · 1992
Earlier work this paper cites.
Bidirectional recurrent neural networks
Mike Schuster and Kuldip K. Paliwal. 1997 · 1997
Earlier work this paper cites.
Introduction to reinforcement learning
Richard S. Sutton and Andrew G. Barto. 1998 · 1998
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015 · 2015
Earlier work this paper cites.
Sequence level training with recurrent neural networks
Original
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016 · 2016
Earlier work this paper cites.
Abstract syntax networks for code generation and semantic parsing
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017 · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
A syntactic neural model for general-purpose code generation
Pengcheng Yin and Graham Neubig. 2017 · 2017
Earlier work this paper cites.
Tree-to-tree neural networks for program translation
Xinyun Chen, Chang Liu, and Dawn Song. 2018 · 2018
Earlier work this paper cites.
Fast abstractive summarization with reinforce-selected sentence rewriting
Yen-Chun Chen and Mohit Bansal. 2018 · 2018
Earlier work this paper cites.

Similar

A retrieve-and-edit framework for predicting structured outputs
Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. 2018 · 2018
Cited alongside, same era.
Code completion with neural attention and pointer networks
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018 · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford and Karthik Narasimhan. 2018 · 2018
Cited alongside, same era.
Improving automatic source code summarization via deep reinforcement learning
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018 · 2018
Cited alongside, same era.
Transformer-xl: Attentive language models beyond a fixed-length context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019 · 2019
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Spoc: Search-based pseudocode to code
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. 2019 · 2019
Cited alongside, same era.
Learning to infer program sketches
Maxwell I. Nye, Luke B. Hewitt, Joshua B. Tenenbaum, and Armando Solar-Lezama. 2019 · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Cited alongside, same era.
Code generation as a dual task of code summarization
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019 · 2019
Cited alongside, same era.
Coacor: Code annotation for code retrieval with reinforcement learning
Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019 · 2019
Cited alongside, same era.
BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.

Then

Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Later among the works it cites.
Intellicode compose: code generation using transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020 · 2020
Later among the works it cites.
Graph-based, self-supervised program repair from diagnostic feedback
Michihiro Yasunaga and Percy Liang. 2020 · 2020
Later among the works it cites.
Unified pre-training for program understanding and generation
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021
Later among the works it cites.
Code prediction by feeding trees to transformers
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021 · 2021
Later among the works it cites.
Energy-based models for code generation under compilability constraints
Tomasz Korbak, Hady ElSahar, Marc Dymetman, and Germán Kruszewski. 2021 · 2021
Later among the works it cites.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021 · 2021
Later among the works it cites.
Retrieval augmented code generation and summarization
Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021
Later among the works it cites.
Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. 2021 · 2021
Later among the works it cites.
Servicebert: A pre-trained model for web service tagging and recommendation
Xin Wang, Pingyi Zhou, Yasheng Wang, Xiao Liu, Jin Liu, and Hao Wu. 2021a · 2021
Later among the works it cites.
Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021b · 2021
Later among the works it cites.
Break-it-fix-it: Unsupervised learning for program repair
Michihiro Yasunaga and Percy Liang. 2021 · 2021
Later among the works it cites.
Reinforcement-learning-guided source code summarization using hierarchical attention
Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, Zhou Zhao, Jian Wu, Philip S. Yu, and Guandong Xu. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…