2021

Retrieval Augmented Code Generation and Summarization

Parvez, Md Rizwan, Ahmad, Wasi Uddin, Chakraborty, Saikat et al.

Understand

Software developers write a lot of source code and documentation during software development.

Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them.
To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models.
REDCODER has a couple of uniqueness.

Built on

Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Tree-transformer: A transformer-based method for correction of tree-structured data
Original
Jacob Harer, Chris Reale, and Peter Chin. 2019 · 1908
Earlier work this paper cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 1909
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
ORANGE: a method for evaluating automatic evaluation metrics for machine translation
Chin-Yew Lin and Franz Josef Och. 2004 · 2004
Earlier work this paper cites.
Xml retrieval
CD Manning, P Raghavan, and H Schütze. 2008 · 2008
Earlier work this paper cites.
Codebleu: a method for automatic evaluation of code synthesis
Original
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009
Earlier work this paper cites.
The probabilistic relevance framework: BM25 and beyond
Stephen Robertson and Hugo Zaragoza. 2009 · 2009
Earlier work this paper cites.
Example-centric programming: integrating web search into the development environment
Joel Brandt, Mira Dontcheva, Marcos Weskamp, and Scott R Klemmer. 2010 · 2010
Earlier work this paper cites.
How well do search engines support code retrieval on the web?
Susan Elliott Sim, Medha Umarji, Sukanya Ratanotayanon, and Cristina V Lopes. 2011 · 2011
Earlier work this paper cites.
Learning discriminative projections for text similarity measures
Wen-tau Yih, Kristina Toutanova, John C Platt, and Christopher Meek. 2011 · 2011
Earlier work this paper cites.
What makes a good code example?: A study of programming q&a in stackoverflow
Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012 · 2012
Earlier work this paper cites.
What help do developers seek, when and how?
Hongwei Li, Zhenchang Xing, Xin Peng, and Wenyun Zhao. 2013 · 2013
Earlier work this paper cites.
Mining stackoverflow to turn the ide into a self-confident programming prompter
Luca Ponzanelli, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Michele Lanza. 2014 · 2014
Earlier work this paper cites.
Solving the search for source code
Kathryn T Stolee, Sebastian Elbaum, and Daniel Dobos. 2014 · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014 · 2014
Earlier work this paper cites.
Source code retrieval on stackoverflow using lda
Achmad Arwan, Siti Rochimah, and Rizky Januar Akbar. 2015 · 2015
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015 · 2015
Earlier work this paper cites.
How developers search for code: a case study
Caitlin Sadowski, Kathryn T Stolee, and Sebastian Elbaum. 2015 · 2015
Earlier work this paper cites.
Deep api learning
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016 · 2016
Earlier work this paper cites.

Similar

Quantization based fast inner product search
Ruiqi Guo, Sanjiv Kumar, Krzysztof Choromanski, and David Simcha. 2016 · 2016
Cited alongside, same era.
Abstract syntax networks for code generation and semantic parsing
Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
What do developers search for on the web?
Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017 · 2017
Cited alongside, same era.
A syntactic neural model for general-purpose code generation
Pengcheng Yin and Graham Neubig. 2017 · 2017
Cited alongside, same era.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018 · 2018
Cited alongside, same era.
A retrieve-and-edit framework for predicting structured outputs
Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018 · 2018
Cited alongside, same era.
Retrieval-based neural code generation
Shirley Anugrah Hayati, Raphael Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig. 2018 · 2018
Cited alongside, same era.
Software search is not a science, even among scientists: A survey of how scientists and engineers find software
Michael Hucka and Matthew J Graham. 2018 · 2018
Cited alongside, same era.
Mapping language to code in programmatic context
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018 · 2018
Cited alongside, same era.
Automatic generation of text descriptive comments for code blocks
Yuding Liang and Kenny Qili Zhu. 2018 · 2018
Cited alongside, same era.
Building language models for text with named entities
Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018 · 2018
Cited alongside, same era.

Then

Code generation as a dual task of code summarization
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019 · 2019
Later among the works it cites.
A transformer-based approach for source code summarization
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020 · 2020
Later among the works it cites.
ELECTRA: Pre-training text encoders as discriminators rather than generators
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020 · 2020
Later among the works it cites.
CodeBERT: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020a · 2020
Later among the works it cites.
CodeBERT: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020b · 2020
Later among the works it cites.
Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020 · 2020
Later among the works it cites.
Dense passage retrieval for open-domain question answering
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020
Later among the works it cites.
Retrieval-augmented generation for knowledge-intensive nlp tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020 · 2020
Later among the works it cites.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020 · 2020
Later among the works it cites.
Incorporating external knowledge through pre-training for natural language to code generation
Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020 · 2020
Later among the works it cites.
Retrieval-based neural source code summarization
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020 · 2020
Later among the works it cites.
Unified pre-training for program understanding and generation
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021
Closest in time.
Graphcodebert: Pre-training code representations with data flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Jian Yin, Daxin Jiang, et al. 2021 · 2021
Closest in time.
Leveraging passage retrieval with generative models for open domain question answering
Gautier Izacard and Edouard Grave. 2021 · 2021
Closest in time.
Retrieval-augmented generation for code summarization via hybrid {gnn}
Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021 · 2021
Closest in time.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021 · 2021
Closest in time.
In-ide code generation from natural language: Promise and challenges
Original
Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2021 · 2021
Closest in time.
Summarizing source code using a neural attention model
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016 · 2083
Closest in time.
A convolutional attention network for extreme summarization of source code
Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. 2016 · 2091
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…