2022

On the Importance of Building High-quality Training Datasets for Neural Code Search

Sun, Zhensu, Li, Li, Liu, Yan et al.

Understand

The performance of neural code search is significantly influenced by the quality of the training data from which the neural models are derived.

A large corpus of high-quality query and code pairs is demanded to establish a precise mapping from the natural language to the programming language.
Due to the limited availability, most widely-used code search datasets are established with compromise, such as using code comments as a replacement of queries.
Our empirical study on a famous code search dataset reveals that over one-third of its queries contain noises that make them deviate from natural user queries.

Built on

A Coefficient of Agreement for Nominal Scales
Cohen and J. 1960 · 1960
Earlier work this paper cites.
Sampling techniques
William G Cochran. 1977 · 1977
Earlier work this paper cites.
Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper
A. Dempster, N. Laird, and D. Rubin. 1977 · 1977
Earlier work this paper cites.
Principal component analysis
S. Wold, K. Esbensen, and P. Geladi. 1987 · 1987
Earlier work this paper cites.
Determining sample size
Glenn D Israel. 1992 · 1992
Earlier work this paper cites.
What makes a good code example?: A study of programming Q & \& A in StackOverflow
Seyed Mehdi Nasehi, Jonathan Sillito, F. Maurer, and C. Burns. 2012 · 2012
Earlier work this paper cites.
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Original
Kyunghyun Cho, B. V. Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Auto-Encoding Variational Bayes
Original
Diederik P. Kingma and M. Welling. 2014 · 2014
Earlier work this paper cites.
Variational Autoencoder based Anomaly Detection using Reconstruction Probability
Jinwon An and S. Cho. 2015 · 2015
Earlier work this paper cites.
Unsupervised change detection using Spatial Transformer Networks
Dan Chianucci and A. Savakis. 2016 · 2016
Earlier work this paper cites.
Accessing Inaccessible Android APIs: An Empirical Study. In The 32nd International Conference on Software Maintenance and Evolution (ICSME 2016)
Li Li, Tegawendé F Bissyandé, Yves Le Traon, and Jacques Klein. 2016 · 2016
Earlier work this paper cites.
Echo-state conditional variational autoencoder for anomaly detection
Suwon Suh, Daniel H. Chae, Hyon-Goo Kang, and S. Choi. 2016 · 2016
Earlier work this paper cites.
CSNIPPEX: automated synthesis of compilable code snippets from Q&A sites
Valerio Terragni, Yepang Liu, and S. C. Cheung. 2016 · 2016
Earlier work this paper cites.
A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation
Original
Antonio Valerio Miceli Barone and Rico Sennrich. 2017 · 2017
Earlier work this paper cites.
Fake Twitter followers detection by denoising autoencoder
Jacopo Castellini, V. Poggioni, and Giulia Sorbi. 2017 · 2017
Earlier work this paper cites.
Finding Anomalies with Generative Adversarial Networks for a Patrolbot
W. Lawson, Esube Bekele, and K. Sullivan. 2017 · 2017
Earlier work this paper cites.
Libraries.io Open Source Repository and Dependency Metadata
A. Nesbitt and Benjamin Nickolls. 2017 · 2017
Earlier work this paper cites.
Anomaly Detection for Temporal Data using Long Short-Term Memory (LSTM)
A. Singh. 2017 · 2017
Earlier work this paper cites.
Detecting rumors on Online Social Networks using multi-layer autoencoder
Y. Zhang, Weiling Chen, C. Yeo, C. Lau, and B. Lee. 2017 · 2017
Earlier work this paper cites.

Similar

A Neural Framework for Retrieval and Summarization of Source Code
Q. Chen and Minghui Zhou. 2018 · 2018
Cited alongside, same era.
Deep Code Search
Xiaodong Gu, H. Zhang, and S. Kim. 2018 · 2018
Cited alongside, same era.
Deep Code Comment Generation
X. Hu, G. Li, Xin Xia, D. Lo, and Zhi Jin. 2018 · 2018
Cited alongside, same era.
FaCoY - A Code-to-Code Search Engine. In The 40th International Conference on Software Engineering (ICSE 2018)
Kisub Kim, Dongsun Kim, Tegawendé F Bissyandé, Eunjong Choi, Li Li, Jacques Klein, and Yves Le Traon. 2018 · 2018
Cited alongside, same era.
Characterising Deprecated Android APIs. In The 15th International Conference on Mining Software Repositories (MSR 2018)
Li Li, Jun Gao, Tegawendé F Bissyandé, Lei Ma, Xin Xia, and Jacques Klein. 2018 · 2018
Cited alongside, same era.
Distributed Anomaly Detection Using Autoencoder Neural Networks in WSN for IoT
Tie Luo and Sai Ganesh Nagarajan. 2018 · 2018
Cited alongside, same era.
An introduction to neural information retrieval
Bhaskar Mitra, Nick Craswell, et al · 2018
Cited alongside, same era.
Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications
Haowen Xu, Wenxiao Chen, N. Zhao, Z. Li, Jiahao Bu, Zhihan Li, Y. Liu, Y. Zhao, D. Pei, Y. Feng, Jian Jhen Chen, Zhaogang Wang, and Honglin Qiao. 2018 · 2018
Cited alongside, same era.
StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow
Ziyu Yao, Daniel S. Weld, W. Chen, and Huan Sun. 2018 · 2018
Cited alongside, same era.
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018
Cited alongside, same era.
Are Code Examples on an Online Q&A Forum Reliable?: A Study of API Misuse on Stack Overflow
Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim. 2018 · 2018
Cited alongside, same era.
When deep learning met code search
José Cambronero, Hongyu Li, S. Kim, K. Sen, and S. Chandra. 2019 · 2019
Cited alongside, same era.

Then

Neural joint attention code search over structure embeddings for software Q & \& A sites
Gang Hu, Min Peng, Yihan Zhang, Qianqian Xie, and Mengting Yuan. 2020 · 2020
Later among the works it cites.
CDA: Characterising Deprecated Android APIs
Li Li, Jun Gao, Tegawendé F Bissyandé, Lei Ma, Xin Xia, and Jacques Klein. 2020a · 2020
Later among the works it cites.
Learning Code-Query Interaction for Enhancing Code Searches
W. Li, Haozhe Qin, Shuhan Yan, Beijun Shen, and Y. Chen. 2020b · 2020
Later among the works it cites.
Anomaly Detection for Time Series Using VAE-LSTM Hybrid Model
Shuyu Lin, R. Clark, R. Birke, Sandro Schönborn, Niki Trigoni, and S. Roberts. 2020 · 2020
Later among the works it cites.
Adaptive Deep Code Search
Chunyang Ling, Zeqi Lin, Yanzhen Zou, and Bing Xie. 2020 · 2020
Later among the works it cites.
Opportunities and Challenges in Code Search Tools
Original
C. Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and J. Grundy. 2020a · 2020
Later among the works it cites.
Simplifying Deep-Learning-Based Model for Code Search
Original
Chao Liu, Xin Xia, David Lo, Zhiwei Liu, A. Hassan, and Shanping Li. 2020b · 2020
Later among the works it cites.
OCoR: An Overlapping-Aware Code Retriever
Zhu Qihao, Sun Ze-yu, Liang Xiran, Xiong Yingfei, and Z. Lu. 2020 · 2020
Later among the works it cites.
Improving Code Search with Co-Attentive Representation Learning
Jianhang Shuai, Ling Xu, Chao Liu, Meng Yan, Xin Xia, and Yan Lei. 2020 · 2020
Later among the works it cites.
PSCS: A Path-based Neural Model for Semantic Code Search
Original
Zhensu Sun, Yan Liu, Chen Yang, and Yu Qian. 2020 · 2020
Later among the works it cites.
TranS3̂: A Transformer-based Framework for Unifying Code Summarization and Code Search
Original
W. Wang, Y. Zhang, Zhengran Zeng, and Guandong Xu. 2020 · 2020
Later among the works it cites.
Are the Code Snippets What We Are Searching for? A Benchmark and an Empirical Study on Code Search with Natural-Language Queries
Shuhan Yan, H. Yu, Y. Chen, Beijun Shen, and L. Jiang. 2020 · 2020
Later among the works it cites.
Leveraging Code Generation to Improve Code Retrieval and Summarization via Dual Learning
Wei Ye, Rui Xie, Jing lei Zhang, Tian xiang Hu, Xiaoyin Wang, and Shikun Zhang. 2020 · 2020
Later among the works it cites.
How to Write Doc Comments for the Javadoc Tool
2021 · 2021
Later among the works it cites.
On the Importance of Building High-quality Training Datasets for Neural Code Search
2021 · 2021
Later among the works it cites.
StackOverflow
2021 · 2021
Later among the works it cites.
Identifying and Characterizing Silently-Evolved Methods in the Android API. In The 43rd ACM/IEEE International Conference on Software Engineering, SEIP Track (ICSE-SEIP 2021)
Pei Liu, Li Li, Yichun Yan, Mattia Fazzini, and John Grundy. 2021 · 2021
Later among the works it cites.
On the Impact of Sample Duplication in Machine Learning based Android Malware Detection
Yanjie Zhao, Li Li, Haoyu Wang, Haipeng Cai, Tegawende Bissyande, Jacques Klein, and John Grundy. 2021 · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…