2021

In-IDE Code Generation from Natural Language: Promise and Challenges

Xu, Frank F., Vasilescu, Bogdan, Neubig, Graham

Understand

A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs.

One major difficulty of programming is turning concept into code, especially when dealing with the APIs of unfamiliar libraries.
Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries, but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested.
We perform the first comprehensive investigation of the promise and challenges of using such technology inside the IDE, asking "at the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?" We first develop a plugin for the IDE that implements a hybrid of code generation and code retrieval functionality, and orchestrate virtual environments to enable collection of many user events.

Built on

Classes of recursively enumerable sets and their decision problems
Henry Gordon Rice. 1953 · 1953
Earlier work this paper cites.
Estimates of location based on rank tests
Joseph L Hodges Jr and Erich L Lehmann. 1963 · 1963
Earlier work this paper cites.
Automatic programming through natural language dialogue: A survey
George E. Heidorn. 1976 · 1976
Earlier work this paper cites.
A Complexity Measure
T. McCabe. 1976 · 1976
Earlier work this paper cites.
Natural Language Processing in an Automatic Programming Domain
J. Ginsparg. 1978 · 1978
Earlier work this paper cites.
On the pooling of time series and cross section data
Yair Mundlak. 1978 · 1978
Earlier work this paper cites.
On the foolishness of “natural language programming”
Edsger W Dijkstra. 1979 · 1979
Earlier work this paper cites.
Watch what I do: programming by demonstration
A. Cypher, Daniel C. Halbert, D. Kurlander, H. Lieberman, D. Maulsby, B. Myers, and Alan Turransky. 1993 · 1993
Earlier work this paper cites.
Query reformulation for dynamic information integration
Yigal Arens, Craig A Knoblock, and Wei-Min Shen. 1996 · 1996
Earlier work this paper cites.
Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence . 1050–1055
John M Zelle and Raymond J Mooney. 1996 · 1996
Earlier work this paper cites.
Mathematical methods of statistics . Vol. 43
Harald Cramér. 1999 · 1999
Earlier work this paper cites.
NaturalJava: a natural language interface for programming in Java. In International Conference on Intelligent User Interfaces (IUI) . 207–211
David Price, Ellen Rilofff, Joseph Zachary, and Brandon Harvey. 2000 · 2000
Earlier work this paper cites.
Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Applied multiple regression/correlation analysis for the behavioral sciences
J. Cohen. 2003 · 2003
Earlier work this paper cites.
Synthesis of Programs in Computational Logic. In Program Development in Computational Logic
D. Basin, Y. Deville, P. Flener, A. Hamfelt, and Jørgen Fischer Nilsson. 2004 · 2004
Earlier work this paper cites.
Designing the whyline: a debugging interface for asking questions about program behavior. In CHI 2004 Conference on Human Factors in Computing Systems (CHI)
A. Ko and B. Myers. 2004 · 2004
Earlier work this paper cites.
Six learning barriers in end-user programming systems. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE, 199–206
Amy Ko, Brad A Myers, and Htet Htet Aung. 2004 · 2004
Earlier work this paper cites.
Natural programming languages and environments
B. Myers, J. Pane, and A. Ko. 2004 · 2004
Earlier work this paper cites.
Programming with unrestricted natural language. In Proceedings of the Australasian Language Technology Workshop 2005 . 191–199
David Vadas and James R Curran. 2005 · 2005
Earlier work this paper cites.
Data analysis using regression and multilevel/hierarchical models
Andrew Gelman and Jennifer Hill. 2006 · 2006
Earlier work this paper cites.
End-User Development: An Emerging Paradigm. In End User Development
H. Lieberman, F. Paternò, Markus Klann, and V. Wulf. 2006 · 2006
Earlier work this paper cites.
NLP (natural language processing) for NLP (natural language programming). In International Conference on Intelligent Text Processing and Computational Linguistics . Springer, 319–330
Rada Mihalcea, Hugo Liu, and Henry Lieberman. 2006 · 2006
Earlier work this paper cites.
Quality, productivity and economic benefits of software reuse: a review of industrial studies
Parastoo Mohagheghi and Reidar Conradi. 2007 · 2007
Earlier work this paper cites.
Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th International Conference on World Wide Web . 521–530
Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007 · 2007
Earlier work this paper cites.
Guide to advanced empirical software engineering
Forrest Shull, Janice Singer, and Dag IK Sjøberg. 2007 · 2007
Earlier work this paper cites.
Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) . 678–687
Luke Zettlemoyer and Michael Collins. 2007 · 2007
Earlier work this paper cites.
Debugging reinvented
A. Ko and B. Myers. 2008 · 2008
Earlier work this paper cites.
Introduction to information retrieval
Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. 2008 · 2008
Earlier work this paper cites.
Program Synthesis by Sketching
Armando Solar-Lezama. 2008 · 2008
Earlier work this paper cites.
Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI)
J. Brandt, P. Guo, J. Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009 · 2009
Earlier work this paper cites.
The Scratch Programming Language and Environment
J. Maloney, M. Resnick, N. Rusk, B. Silverman, and Evelyn Eastmond. 2010 · 2010
Earlier work this paper cites.
Automating string processing in spreadsheets using input-output examples
Sumit Gulwani. 2011 · 2011
Earlier work this paper cites.
Harnessing Stack Overflow for the IDE. In International Workshop on Recommendation Systems for Software Engineering (RSSE) . IEEE, 26–30
Alberto Bacchelli, Luca Ponzanelli, and Michele Lanza. 2012 · 2012
Earlier work this paper cites.
Lateral collinearity and misleading results in variance-based SEM: An illustration and recommendations
Ned Kock and Gary Lynn. 2012 · 2012
Earlier work this paper cites.
Experimentation in software engineering
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012 · 2012
Earlier work this paper cites.
Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP) . 1533–1544
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013 · 2013
Earlier work this paper cites.
Rubric based assessment plan implementation for Computer Science program: A practical approach
M. Dawood, Khalid A. Buragga, Abdul Raouf Khan, and Noor Zaman. 2013 · 2013
Earlier work this paper cites.
Automatic query reformulations for text retrieval in software engineering
Sonia Haiduc, G. Bavota, A. Marcus, R. Oliveto, A. Lucia, and T. Menzies. 2013 · 2013
Earlier work this paper cites.
Using Semantic Unification to Generate Regular Expressions from Natural Language. In The 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL)
Nate Kushman and R. Barzilay. 2013 · 2013
Earlier work this paper cites.
From Natural Language Specifications to Program Input Parsers. In The 51st Annual Meeting of the Association for Computational Linguistics (ACL)
Tao Lei, F. Long, R. Barzilay, and M. Rinard. 2013 · 2013
Earlier work this paper cites.
Integrating Programming by Example and Natural Language Programming. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)
Mehdi Manshadi, Daniel Gildea, and James F. Allen. 2013 · 2013
Earlier work this paper cites.
A general and simple method for obtaining R2 from generalized linear mixed-effects models
Shinichi Nakagawa and Holger Schielzeth. 2013 · 2013
Earlier work this paper cites.
Seahawk: Stack Overflow in the IDE. In International Conference on Software Engineering (ICSE) . IEEE, 1295–1298
Luca Ponzanelli, Alberto Bacchelli, and Michele Lanza. 2013 · 2013
Earlier work this paper cites.

Similar

Learning natural coding conventions. In International Symposium on Foundations of Software Engineering (ESEC/FSE) . 281–293
Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2014 · 2014
Cited alongside, same era.
NL-based query refinement and contextualized code search results: A user study
E. Hill, Manuel Roldan-Vega, J. Fails, and Greg Mallet. 2014 · 2014
Cited alongside, same era.
Extension of Nakagawa & Schielzeth’s R G L M M 2 R^{2}_{GLMM} to random slopes models
Paul CD Johnson. 2014 · 2014
Cited alongside, same era.
Spotting working code examples. In 36th International Conference on Software Engineering (ICSE)
I. Keivanloo, J. Rilling, and Ying Zou. 2014 · 2014
Cited alongside, same era.
FlashExtract: a framework for data extraction by examples
Vu Le and Sumit Gulwani. 2014 · 2014
Cited alongside, same era.
Mining Stack Overflow to turn the IDE into a self-confident programming prompter. In International Conference on Mining Software Repositories (MSR)
Luca Ponzanelli, G. Bavota, M. D. Penta, R. Oliveto, and M. Lanza. 2014 · 2014
Cited alongside, same era.
SurfClipse: Context-Aware Meta-search in the IDE
M. M. Rahman and C. Roy. 2014 · 2014
Cited alongside, same era.
Towards a context-aware IDE-based meta search engine for recommendation about programming errors and exceptions. In International Conference on Software Analysis, Evolution, and Reengineering (SANER) . IEEE, 194–203
Mohammad Masudur Rahman, Shamima Yeasmin, and Chanchal K Roy. 2014 · 2014
Cited alongside, same era.
Code completion with statistical language models. In ACM Conference on Programming Language Design and Implementation (PLDI) . ACM, 419–428
Veselin Raychev, Martin Vechev, and Eran Yahav. 2014 · 2014
Cited alongside, same era.
Live API documentation
Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014 · 2014
Cited alongside, same era.
On the localness of software. In International Symposium on Foundations of Software Engineering (ESEC/FSE) . ACM, 269–280
Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014 · 2014
Cited alongside, same era.
Information extraction over structured data: Question answering with freebase. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) . 956–966
Xuchen Yao and Benjamin Van Durme. 2014 · 2014
Cited alongside, same era.

Then

Thematic analysis: Striving to meet the trustworthiness criteria
Lorelli S Nowell, Jill M Norris, Deborah E White, and Nancy J Moules. 2017 · 2017
Later among the works it cites.
Neuro-Symbolic Program Synthesis
Emilio Parisotto, Abdel rahman Mohamed, R. Singh, L. Li, Dengyong Zhou, and Pushmeet Kohli. 2017 · 2017
Later among the works it cites.
ANNE: Improving Source Code Search using Entity Retrieval Approach. In WSDM ’17
Venkatesh Vinayakarao, A. Sarma, R. Purandare, Shuktika Jain, and Saumya Jain. 2017 · 2017
Later among the works it cites.
A syntactic neural model for general-purpose code generation
Pengcheng Yin and Graham Neubig. 2017 · 2017
Later among the works it cites.
Seq2sql: Generating structured queries from natural language using reinforcement learning
Original
Victor Zhong, Caiming Xiong, and Richard Socher. 2017 · 2017
Later among the works it cites.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018 · 2018
Later among the works it cites.
pix2code: Generating Code from a Graphical User Interface Screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2018, Paris, France, June 19-22, 2018 . ACM, 3:1–3:6
Tony Beltramelli. 2018 · 2018
Later among the works it cites.
Rousillon: Scraping Distributed Hierarchical Web Data
Sarah E. Chasins, Maria Mueller, and Rastislav Bodík. 2018 · 2018
Later among the works it cites.
Program synthesis using conflict-driven learning
Y. Feng, R. Martins, Osbert Bastani, and Isil Dillig. 2018 · 2018
Later among the works it cites.
What We Can Learn About Student Learning From Open-Ended Programming Projects in Middle School Computer Science
Shuchi Grover, S. Basu, and Patricia K. Schank. 2018 · 2018
Later among the works it cites.
Deep code search. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) . IEEE, 933–944
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018 · 2018
Later among the works it cites.
A retrieve-and-edit framework for predicting structured outputs
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018 · 2018
Later among the works it cites.
Retrieval-Based Neural Code Generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, Brussels, Belgium, 925–930
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig. 2018 · 2018
Later among the works it cites.
Interactive Extraction of Examples from Existing Code
Andrew Head, Elena Leah Glassman, B. Hartmann, and Marti A. Hearst. 2018 · 2018
Later among the works it cites.
Mapping Language to Code in Programmatic Context. In 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Srini Iyer, Ioannis Konstas, A. Cheung, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, I. Labutov, X. Li, X. Zhang, W. Shi, Wanling Ding, Tom Michael Mitchell, and B. Myers. 2018 · 2018
Later among the works it cites.
API designers in the field: Design practices and challenges for creating usable APIs. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE, 249–258
Lauren Murphy, Mary Beth Kery, Oluwatosin Alliyu, Andrew Macvean, and Brad A Myers. 2018 · 2018
Later among the works it cites.
Enriched event streams: a general dataset for empirical studies on in-IDE activities of software developers. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR) . 62–65
Sebastian Proksch, Sven Amann, and Sarah Nadi. 2018 · 2018
Later among the works it cites.
StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
Ziyu Yao, Daniel S. Weld, W. Chen, and Huan Sun. 2018 · 2018
Later among the works it cites.
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow. In International Conference on Mining Software Repositories (MSR) . ACM, 476–486
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018
Later among the works it cites.
Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation
Pengcheng Yin and Graham Neubig. 2018 · 2018
Later among the works it cites.
NAPS: Natural program synthesis dataset
Maksym Zavershynskyi, Alex Skidanov, and Illia Polosukhin. 2018 · 2018
Later among the works it cites.
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP/IJCNLP)
R. Agashe, Srini Iyer, and Luke Zettlemoyer. 2019 · 2019
Later among the works it cites.
Fixed and random effects models: making an informed choice
Andrew Bell, Malcolm Fairbrother, and Kelvyn Jones. 2019 · 2019
Later among the works it cites.
Execution-Guided Neural Program Synthesis. In 7th International Conference on Learning Representations (ICLR)
X. Chen, C. Liu, and D. Song. 2019 · 2019
Later among the works it cites.
Write, Execute, Assess: Program Synthesis with a REPL. In 33rd Conference on Neural Information Processing Systems (NeurIPS)
K. Ellis, Maxwell Nye, Y. Pu, Felix Sosa, J. Tenenbaum, and Armando Solar-Lezama. 2019 · 2019
Later among the works it cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 2019
Later among the works it cites.
SPoC: Search-based Pseudocode to Code. In 33rd Conference on Neural Information Processing Systems (NeurIPS)
S. Kulal, Panupong Pasupat, K. Chandra, Mina Lee, Oded Padon, A. Aiken, and Percy Liang. 2019 · 2019
Later among the works it cites.
PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, J. Jia, Kirielle Singarajah, Tom Michael Mitchell, and B. Myers. 2019 · 2019
Later among the works it cites.
Marble: Mining for boilerplate code to identify API usability problems. In International Conference on Automated Software Engineering (ASE) . IEEE, 615–627
Daye Nam, Amber Horvath, Andrew Macvean, Brad Myers, and Bogdan Vasilescu. 2019 · 2019
Later among the works it cites.
Program Synthesis and Semantic Parsing with Learned Code Idioms
Richard Shin, Miltiadis Allamanis, Marc Brockschmidt, and Oleksandr Polozov. 2019 · 2019
Later among the works it cites.
Does BLEU score work for code migration?. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 165–176
Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019 · 2019
Later among the works it cites.
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning
Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019b · 2019
Later among the works it cites.
Reranking for Neural Semantic Parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, Florence, Italy, 4553–4559
Pengcheng Yin and Graham Neubig. 2019 · 2019
Later among the works it cites.
CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Zhangyin Feng, Daya Guo, Duyu Tang, N. Duan, X. Feng, Ming Gong, Linjun Shou, B. Qin, Ting Liu, Daxin Jiang, and M. Zhou. 2020 · 2020
Later among the works it cites.
Learning Adaptive Language Interfaces through Decomposition. In Proceedings of the First Workshop on Interactive and Executable Semantic Parsing . Association for Computational Linguistics, Online, 23–33
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. 2020 · 2020
Later among the works it cites.
Opportunities and Challenges in Code Search Tools
Original
C. Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and J. Grundy. 2020 · 2020
Later among the works it cites.
ColloQL: Robust Text-to-SQL Over Search Queries. In Proceedings of the First Workshop on Interactive and Executable Semantic Parsing . 34–45
Karthik Radhakrishnan, Arvind Srikantan, and Xi Victoria Lin. 2020 · 2020
Later among the works it cites.
DeepTC-Enhancer: Improving the Readability of Automatically Generated Tests. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 287–298
Devjeet Roy, Ziyi Zhang, Maggie Ma, Venera Arnaoudova, Annibale Panichella, Sebastiano Panichella, Danielle Gonzalez, and Mehdi Mirakhorli. 2020 · 2020
Later among the works it cites.
Supporting the understanding and comparison of low-code development platforms
Apurvanand Sahay, Arsene Indamutsa, D. D. Ruscio, and A. Pierantonio. 2020 · 2020
Later among the works it cites.
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation. In Annual Meeting of the Association for Computational Linguistics (ACL) . Association for Computational Linguistics, 6045–6052
Frank F Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020 · 2020
Later among the works it cites.
Semantic Scaffolds for Pseudocode-to-Code Generation. In ACL
Ruiqi Zhong, Mitchell Stern, and D. Klein. 2020 · 2020
Later among the works it cites.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Original
William Fedus, Barret Zoph, and Noam Shazeer. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…