Understand
Recent advancements in natural language processing \cite{gpt2} \cite{BERT} have led to near-human performance in multiple natural language tasks.
- In this paper, we seek to understand whether similar techniques can be applied to a highly structured environment with strict syntax rules.
- Specifically, we propose an end-to-end machine learning model for code generation in the Python language built on-top of pre-trained language models.
- We demonstrate that a fine-tuned model can perform well in code generation tasks, achieving a BLEU score of 0.22, an improvement of 46\% over a reasonable sequence-to-sequence baseline.
Built on
On the properties of neural machine translation: Encoder-decoder approaches
KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio · 2014
Earlier work this paper cites.
Character-aware neural language models
Yoon Kim, Yacine Jernite, David A. Sontag, and Alexander M. Rush · 2015
Earlier work this paper cites.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles A. Sutton · 2017
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Similar
BERT: pre-training of deep bidirectional transformers for language understanding
Kenton Lee Kristina Toutanova Jacob Devlin, Ming-Wei Chang · 2018
Cited alongside, same era.
Treegan: Syntax-aware sequence generation with generative adversarial networks
Xinyue Liu, Xiangnan Kong, Lei Liu, and Kuorong Chiang · 2018
Cited alongside, same era.
A grammar-based structural CNN decoder for code generation
Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang · 2018
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Cited alongside, same era.
Then
Codesearchnet challenge: Evaluating the state of semantic code search, 2019
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt · 2019
Later among the works it cites.
Codegru: Context-aware deep learning with gated recurrent unit for source code modeling
Yasir Hussain, Zhiqiu Huang, Senzhang Wang, and Yu Zhou · 2019
Later among the works it cites.
Treegen: A tree-based transformer architecture for code generation, 2019
Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang · 2019
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…