2022

CoCoSoDa: Effective Contrastive Learning for Code Search

Shi, Ensheng, Wang, Yanlin, Gu, Wenchao et al.

Understand

Code search aims to retrieve semantically relevant code snippets for a given natural language query.

Recently, many approaches employing contrastive learning have shown promising results on code representation learning and greatly improved the performance of code search.
However, there is still a lot of room for improvement in using contrastive learning for code search.
In this paper, we propose CoCoSoDa to effectively utilize contrastive learning for code search via two key factors in contrastive learning: data augmentation and negative samples.

Built on

P. Jaccard, “Étude comparative de la distribution florale dans une portion des alpes et des jura,” Bull Soc Vaudoise Sci Nat , pp. 547–579, 1901
1901
Earlier work this paper cites.
H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” CoRR , vol. abs/1909.09436, 2019. [Online]. Available: http://arxiv.org/abs/1909.09436
Original
1909
Earlier work this paper cites.
S. E. Robertson and K. S. Jones, “Relevance weighting of search terms,” Journal of the American Society for Information science , pp. 129–146, 1976
1976
Earlier work this paper cites.
J. Singer, T. C. Lethbridge, N. G. Vinson, and N. Anquetil, “An examination of software engineering work practices,” in CASCON . IBM, 1997, p. 21
1997
Earlier work this paper cites.
R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA . IEEE Computer Society, 2006, pp. 1735–1742. [Online]. Available: https://doi.org/10.1109/CVPR.2006.100
2006
Earlier work this paper cites.
H. Schütze, C. D. Manning, and P. Raghavan, Introduction to information retrieval . Cambridge University Press Cambridge, 2008, vol. 39
2008
Earlier work this paper cites.
L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008
2008
Earlier work this paper cites.
S. P. Reiss, “Semantics-based code search,” in ICSE . IEEE, 2009, pp. 243–253
2009
Earlier work this paper cites.
E. Linstead, S. K. Bajracharya, T. C. Ngo, P. Rigor, C. V. Lopes, and P. Baldi, “Sourcerer: mining and searching internet-scale software repositories,” Data Min. Knowl. Discov. , vol. 18, no. 2, pp. 300–336, 2009
2009
Earlier work this paper cites.
M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in ESEC/FSE , 2009, pp. 213–222
2009
Earlier work this paper cites.
C. McMillan, M. Grechanik, D. Poshyvanyk, Q. Xie, and C. Fu, “Portfolio: finding relevant functions and their usage,” in ICSE . ACM, 2011, pp. 111–120
2011
Earlier work this paper cites.
V. Raychev, M. T. Vechev, and E. Yahav, “Code completion with statistical language models,” in PLDI , 2014, pp. 419–428
2014
Earlier work this paper cites.
A. G. Howard, “Some improvements on deep convolutional neural network based image classification,” in ICLR (Poster) , 2014
2014
Earlier work this paper cites.
Y. Kim, “Convolutional neural networks for sentence classification,” in EMNLP . ACL, 2014, pp. 1746–1751
2014
Earlier work this paper cites.
K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in EMNLP . ACL, 2014, pp. 1724–1734
2014
Earlier work this paper cites.
M. Lu, X. Sun, S. Wang, D. Lo, and Y. Duan, “Query expansion via wordnet for effective code search,” in SANER . IEEE Computer Society, 2015, pp. 545–549
2015
Earlier work this paper cites.
F. Lv, H. Zhang, J. Lou, S. Wang, D. Zhang, and J. Zhao, “Codehow: Effective code search based on API understanding and extended boolean model (E),” in ASE . IEEE Computer Society, 2015, pp. 260–270
2015
Earlier work this paper cites.
S. Proksch, J. Lerch, and M. Mezini, “Intelligent code completion with bayesian networks,” ACM Trans. Softw. Eng. Methodol. , vol. 25, no. 1, pp. 3:1–3:31, 2015
2015
Earlier work this paper cites.
M. L. Vásquez, L. F. Cortes-Coy, J. Aponte, and D. Poshyvanyk, “Changescribe: A tool for automatically generating commit messages,” in ICSE (2) . IEEE Computer Society, 2015, pp. 709–712
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in ICML , ser. JMLR Workshop and Conference Proceedings, vol. 37. JMLR.org, 2015, pp. 448–456
2015
Earlier work this paper cites.
M. Iyyer, V. Manjunatha, J. L. Boyd-Graber, and H. D. III, “Deep unordered composition rivals syntactic methods for text classification,” in ACL (1) . The Association for Computer Linguistics, 2015, pp. 1681–1691
2015
Earlier work this paper cites.
L. Nie, H. Jiang, Z. Ren, Z. Sun, and X. Li, “Query expansion based on crowd knowledge for code search,” IEEE Trans. Serv. Comput. , vol. 9, no. 5, pp. 771–783, 2016
2016
Earlier work this paper cites.
S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in ACL , vol. 1. The Association for Computer Linguistics, 2016
2016
Earlier work this paper cites.
T. Devries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” CoRR , vol. abs/1708.04552, 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
F. Zhang, H. Niu, I. Keivanloo, and Y. Zou, “Expanding queries for code search using semantically related API class-names,” IEEE Trans. Software Eng. , vol. 44, no. 11, pp. 1070–1082, 2018
2018
Earlier work this paper cites.
M. Allamanis, E. T. Barr, P. T. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Comput. Surv. , vol. 51, no. 4, pp. 81:1–81:37, 2018
2018
Earlier work this paper cites.
X. Gu, H. Zhang, and S. Kim, “Deep code search,” in ICSE . ACM, 2018, pp. 933–944
2018
Earlier work this paper cites.
S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in ICLR (Poster) . OpenReview.net, 2018
2018
Earlier work this paper cites.

Similar

Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in CVPR . Computer Vision Foundation / IEEE Computer Society, 2018, pp. 3733–3742
2018
Cited alongside, same era.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2018
2018
Cited alongside, same era.
J. Cambronero, H. Li, S. Kim, K. Sen, and S. Chandra, “When deep learning met code search,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019 , M. Dumas, D. Pfahl, S. Apel, and A. Russo, Eds. ACM, 2019, pp. 964–974. [Online]. Available: https://doi.org/10.1145/3338906.3340458
2019
Cited alongside, same era.
Y. Wan, J. Shu, Y. Sui, G. Xu, Z. Zhao, J. Wu, and P. S. Yu, “Multi-modal attention network learning for semantic source code retrieval,” in ASE . IEEE, 2019, pp. 13–25
2019
Cited alongside, same era.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR , vol. abs/1907.11692, 2019
Original
2019
Cited alongside, same era.
A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” in ICSE , 2019, pp. 795–806
2019
Cited alongside, same era.
S. Xu, Y. Yao, F. Xu, T. Gu, H. Tong, and J. Lu, “Commit message generation for source code changes,” in IJCAI . ijcai.org, 2019, pp. 3975–3981
2019
Cited alongside, same era.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR , 2019
2019
Cited alongside, same era.
J. Gao, D. He, X. Tan, T. Qin, L. Wang, and T. Liu, “Representation degeneration problem in training natural language generation models,” in ICLR (Poster) . OpenReview.net, 2019
2019
Cited alongside, same era.
W. Li, H. Qin, S. Yan, B. Shen, and Y. Chen, “Learning code-query interaction for enhancing code searches,” in ICSME . IEEE, 2020, pp. 115–126
2020
Cited alongside, same era.
Q. Zhu, Z. Sun, X. Liang, Y. Xiong, and L. Zhang, “Ocor: An overlapping-aware code retriever,” in ASE , 2020
2020
Cited alongside, same era.
J. Shuai, L. Xu, C. Liu, M. Yan, X. Xia, and Y. Lei, “Improving code search with co-attentive representation learning,” in ICPC ’20: 28th International Conference on Program Comprehension, Seoul, Republic of Korea, July 13-15, 2020 . ACM, 2020, pp. 196–207. [Online]. Available: https://doi.org/10.1145/3387904.3389269
2020
Cited alongside, same era.

Then

D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=jLoC4ez43PZ
2021
Later among the works it cites.
Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in EMNLP (1) . Association for Computational Linguistics, 2021, pp. 8696–8708
2021
Later among the works it cites.
W. U. Ahmad, S. Chakraborty, B. Ray, and K. Chang, “Unified pre-training for program understanding and generation,” in NAACL-HLT . Association for Computational Linguistics, 2021, pp. 2655–2668
2021
Later among the works it cites.
P. Jain, A. Jain, T. Zhang, P. Abbeel, J. Gonzalez, and I. Stoica, “Contrastive code representation learning,” in EMNLP (1) . Association for Computational Linguistics, 2021, pp. 5954–5971
2021
Later among the works it cites.
N. D. Q. Bui, Y. Yu, and L. Jiang, “Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations,” in SIGIR . ACM, 2021, pp. 511–521
2021
Later among the works it cites.
E. Shi, Y. Wang, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “Cast: Enhancing code summarization with hierarchical splitting and reconstruction of abstract syntax trees,” in EMNLP , 2021
2021
Later among the works it cites.
L. Du, X. Shi, Y. Wang, E. Shi, S. Han, and D. Zhang, “Is a single model enough? mucos: A multi-model ensemble learning approach for semantic code search,” in Proceedings of the 30th ACM International Conference on Information & Knowledge Management , 2021, pp. 2994–2998
2021
Later among the works it cites.
Y. Wang and H. Li, “Code completion by modeling flattened abstract syntax trees as graphs,” in AAAI , 2021
2021
Later among the works it cites.
W. Tao, Y. Wang, E. Shi, L. Du, S. Han, H. Zhang, D. Zhang, and W. Zhang, “On the evaluation of commit message generation models: An experimental study,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2021, pp. 126–136
2021
Later among the works it cites.
T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” in EMNLP (1) . Association for Computational Linguistics, 2021, pp. 6894–6910
2021
Later among the works it cites.
J. M. Giorgi, O. Nitski, B. Wang, and G. D. Bader, “Declutr: Deep contrastive learning for unsupervised textual representations,” in ACL/IJCNLP (1) . Association for Computational Linguistics, 2021, pp. 879–895
2021
Later among the works it cites.
X. Wang, Y. Wang, F. Mi, P. Zhou, Y. Wan, X. Liu, L. Li, H. Wu, J. Liu, and X. Jiang, “Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation,” arXiv preprint arXiv:2108.04556 , 2021
Original
2021
Later among the works it cites.
Y. Ding, L. Buratti, S. Pujar, A. Morari, B. Ray, and S. Chakraborty, “Contrastive learning for source code with structural and functional properties,” CoRR , vol. abs/2110.03868, 2021
Original
2021
Later among the works it cites.
W. Gu, Z. Li, C. Gao, C. Wang, H. Zhang, Z. Xu, and M. R. Lyu, “Cradle: Deep code retrieval based on semantic dependency learning,” Neural Networks , vol. 141, pp. 385–394, 2021
2021
Later among the works it cites.
J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, and N. Duan, “Cosqa: 20, 000+ web queries for code search and question answering,” in ACL , 2021
2021
Later among the works it cites.
M. Mosbach, M. Andriushchenko, and D. Klakow, “On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines,” in ICLR . OpenReview.net, 2021
2021
Later among the works it cites.
F. Wang and H. Liu, “Understanding the behaviour of contrastive loss,” in CVPR . Computer Vision Foundation / IEEE, 2021, pp. 2495–2504
2021
Later among the works it cites.
Y. Meng, C. Xiong, P. Bajaj, S. Tiwary, P. Bennett, J. Han, and X. Song, “COCO-LM: correcting and contrasting text sequences for language model pretraining,” in NeurIPS , 2021, pp. 23 102–23 114
2021
Later among the works it cites.
A. Bansal, S. Haque, and C. McMillan, “Project-level encoding for neural source code summarization of subroutines,” in ICPC . IEEE, 2021, pp. 253–264
2021
Later among the works it cites.
D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in ACL (1) . Association for Computational Linguistics, 2022, pp. 7212–7225
2022
Closest in time.
E. Shi, Y. Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in ICSE , 2022
2022
Closest in time.
W. Gu, Y. Wang, L. Du, H. Zhang, S. Han, D. Zhang, and M. R. Lyu, “Accelerating code search with deep hashing and code classification,” in ACL , 2022
2022
Closest in time.
W. Tao, Y. Wang, E. Shi, L. Du, S. Han, H. Zhang, D. Zhang, and W. Zhang, “A large-scale empirical study of commit message generation: models, datasets and evaluation,” Empirical Software Engineering , vol. 27, no. 7, p. 198, 2022
2022
Closest in time.
E. Shi, Y. Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “RACE: Retrieval-augmented commit message generation,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 5520–5530. [Online]. Available: https://aclanthology.org/2022.emnlp-main.372
2022
Closest in time.
C. Niu, C. Li, V. Ng, J. Ge, L. Huang, and B. Luo, “Spt-code: Sequence-to-sequence pre-training for learning source code representations,” in ICSE . ACM, 2022, pp. 1–13
2022
Closest in time.
Nvidia, “Nvidia a100 80gb pcie gpu,” Product Brief , 2022. [Online]. Available: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/PB-10577-001_v02.pdf
2022
Closest in time.
C. Liu, X. Xia, D. Lo, C. Gao, X. Yang, and J. C. Grundy, “Opportunities and challenges in code search tools,” ACM Comput. Surv. , vol. 54, no. 9, pp. 196:1–196:40, 2022
2022
Closest in time.
CoCoSoDa, “Replication package,” ICSE , 2023. [Online]. Available: https://github.com/DeepSoftwareAnalytics/CoCoSoDa
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…