2022

MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages

Wang, Zhiruo, Cuenca, Grace, Zhou, Shuyan et al.

Understand

While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric.

This creates a barrier for program developers who are not proficient in English.
To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English.
Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian.

Built on

Efficient question answering with question decomposition and multiple answer streams
Sven Hartrumpf, Ingo Glöckner, and Johannes Leveling. 2008 · 2008
Earlier work this paper cites.
Sniff: A search engine for java using free-form queries
Shaunak Chatterjee, Sudeep Juvekar, and Koushik Sen. 2009 · 2009
Earlier work this paper cites.
Bootstrap pattern learning for open-domain CLQA
Hideki Shima and Teruko Mitamura. 2010 · 2010
Earlier work this paper cites.
Concealed data poisoning attacks on nlp models
Original
Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. 2020 · 2010
Earlier work this paper cites.
Mining source code descriptions from developer communications
Sebastiano Panichella, Jairo Aponte, Massimiliano Di Penta, Andrian Marcus, and Gerardo Canfora. 2012 · 2012
Earlier work this paper cites.
Example overflow: Using social media for code recommendation
Alexey Zagalsky, Ohad Barzilay, and Amiram Yehudai. 2012 · 2012
Earlier work this paper cites.
Natural language models for predicting programming comments
Dana Movshovitz-Attias and William W. Cohen. 2013 · 2013
Earlier work this paper cites.
Learning multilingual named entity recognition from wikipedia
Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013 · 2013
Earlier work this paper cites.
Autocomment: Mining question and answer sites for automatic comment generation
Edmund Wong, Jinqiu Yang, and Lin Tan. 2013 · 2013
Earlier work this paper cites.
Learning to generate pseudo-code from source code using statistical machine translation
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015 · 2015
Earlier work this paper cites.
Language to code: Learning semantic parsers for if-this-then-that recipes
Chris Quirk, Raymond Mooney, and Michel Galley. 2015 · 2015
Earlier work this paper cites.
Clocom: Mining existing source code for automatic comment generation
Edmund Wong, Taiyue Liu, and Lin Tan. 2015 · 2015
Earlier work this paper cites.
Latent predictor networks for code generation
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. 2016 · 2016
Earlier work this paper cites.
Improving neural machine translation models with monolingual data
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Earlier work this paper cites.
Seq2sql: Generating structured queries from natural language using reinforcement learning
Original
Victor Zhong, Caiming Xiong, and Richard Socher. 2017 · 2016
Earlier work this paper cites.
Abstract syntax networks for code generation and semantic parsing
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017 · 2017
Earlier work this paper cites.
Multilingual bert readme
Jacob Devlin. 2018 · 2018
Earlier work this paper cites.
The hitchhiker’s guide to testing statistical significance in natural language processing
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018 · 2018
Earlier work this paper cites.

Similar

Understanding back-translation at scale
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018 · 2018
Cited alongside, same era.
Summarizing source code with transferred API knowledge
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018 · 2018
Cited alongside, same era.
Mapping language to code in programmatic context
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018 · 2018
Cited alongside, same era.
Staqc: A systematically mined question-code dataset from stack overflow
Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, and Huan Sun. 2018 · 2018
Cited alongside, same era.
Learning to mine aligned code and natural language pairs from stack overflow
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018b · 2018
Cited alongside, same era.
TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation
Pengcheng Yin and Graham Neubig. 2018 · 2018
Cited alongside, same era.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019
Cited alongside, same era.
Generalized data augmentation for low-resource translation
Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019 · 2019
Cited alongside, same era.
A transformer-based approach for source code summarization
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020 · 2020
Cited alongside, same era.
TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020 · 2020
Cited alongside, same era.
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020
Cited alongside, same era.
XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2020
Cited alongside, same era.

Then

Incorporating external knowledge through pre-training for natural language to code generation
Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020 · 2020
Later among the works it cites.
Program synthesis with large language models
Original
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021 · 2021
Later among the works it cites.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021
Later among the works it cites.
Beyond english-centric multilingual machine translation
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021 · 2021
Later among the works it cites.
Measuring coding challenge competence with apps
Original
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021 · 2021
Later among the works it cites.
MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021 · 2021
Later among the works it cites.
Lyra: A benchmark for turducken-style code generation
Original
Qingyuan Liang, Zeyu Sun, Qihao Zhu, Wenjie Zhang, Lian Yu, Yingfei Xiong, and Lu Zhang. 2021 · 2021
Later among the works it cites.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021 · 2021
Later among the works it cites.
Code generation from natural language with less prior knowledge and more monolingual data
Sajad Norouzi, Keyi Tang, and Yanshuai Cao. 2021 · 2021
Later among the works it cites.
An empirical cybersecurity evaluation of github copilot’s code contributions
Original
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021 · 2021
Later among the works it cites.
Multi-domain multilingual question answering
Sebastian Ruder and Avi Sil. 2021 · 2021
Later among the works it cites.
Zero-shot cross-lingual semantic parsing
Original
Tom Sherborne and Mirella Lapata. 2021 · 2021
Later among the works it cites.
Cross-lingual training with dense retrieval for document retrieval
Original
Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2021 · 2021
Later among the works it cites.
CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021 · 2021
Later among the works it cites.
mT5: A massively multilingual pre-trained text-to-text transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021
Later among the works it cites.
The flores-101 evaluation benchmark for low-resource and multilingual machine translation
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022 · 2022
Closest in time.
Summarizing source code using a neural attention model
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016 · 2083
Closest in time.
A convolutional attention network for extreme summarization of source code
Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016 · 2091
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…