2021

Lyra: A Benchmark for Turducken-Style Code Generation

Liang, Qingyuan, Sun, Zeyu, Zhu, Qihao et al.

Understand

Recently, neural techniques have been used to generate source code automatically.

While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages.
Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts.
In this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language.

Built on

Evaluation of spoken language systems: the ATIS domain
Patti J. Price · 1990
Earlier work this paper cites.
Software engineering, 4th Edition
Ian Sommerville · 1992
Earlier work this paper cites.
Expanding the scope of the ATIS task: The ATIS-3 corpus
Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao, Alexander I. Rudnicky, and Elizabeth Shriberg · 1994
Earlier work this paper cites.
Bleu: A method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Learning to generate pseudo-code from source code using statistical machine translation
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura · 2015
Earlier work this paper cites.
Latent predictor networks for code generation
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew W. Senior · 2016
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2016
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Sqlizer: query synthesis from natural language
Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig · 2017
Earlier work this paper cites.

Similar

Seq2sql: Generating structured queries from natural language using reinforcement learning
Original
Victor Zhong, Caiming Xiong, and Richard Socher · 2017
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Cited alongside, same era.
Mapping language to code in programmatic context
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer · 2018
Cited alongside, same era.
Learning to mine aligned code and natural language pairs from stack overflow
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig · 2018
Cited alongside, same era.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev · 2018
Cited alongside, same era.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Cited alongside, same era.
Codebert: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou · 2020
Cited alongside, same era.
Graphcodebert: Pre-training code representations with data flow
Original
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al · 2020
Cited alongside, same era.

Then

Deep learning based program generation from requirements text: Are we there yet?
H. Liu, M. Shen, J. Zhu, N. Niu, G. Li, and L. Zhang · 2020
Later among the works it cites.
Treegen: A tree-based transformer architecture for code generation
Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang · 2020
Later among the works it cites.
Code line generation based on deep context-awareness of onsite programming
Chuanqi Tao, Panpan Bao, and Zhiqiu Huang · 2020
Later among the works it cites.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021
Closest in time.
Transregex: Multi-modal regular expression synthesis by generate-and-repair
Yeting Li, Shuaimin Li, Zhiwu Xu, Jialun Cao, Zixuan Chen, Yun Hu, Haiming Chen, and Shing-Chi Cheung · 2021
Closest in time.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al · 2021
Closest in time.
Sead: End-to-end text-to-sql generation with schema-aware denoising
Original
Kuan Xuan, Yongbo Wang, Yongliang Wang, Zujie Wen, and Yang Dong · 2021
Closest in time.
A survey of deep learning based text-to-sql generation (in chinese)
Qingyuan Liang, Qihao Zhu, Zeyu Sun, Lu Zhang, Wenjie Zhang, Yingfei Xiong, Guangtai Liang, and Lian Yu · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…