2022

CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training

Wang, Xin, Wang, Yasheng, Wan, Yao et al.

Understand

Recent years have witnessed increasing interest in code representation learning, which aims to represent the semantics of source code into distributed vectors.

Currently, various works have been proposed to represent the complex semantics of source code from different views, including plain text, Abstract Syntax Tree (AST), and several kinds of code graphs (e.g., Control/Data Flow Graph).
However, most of them only consider a single view of source code independently, ignoring the correspondences among different views.
In this paper, we propose to integrate different views with the natural-language description of source code into a unified framework with Multi-View contrastive Pre-training, and name our model as CODE-MVP.

Built on

Roberta: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 1909
Earlier work this paper cites.
Attribute grammar paradigms - A high-level methodology in language implementation
Jukka Paakki. 1995 · 1995
Earlier work this paper cites.
Programl: Graph-based deep learning for program optimization and analysis
Original
Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun, Torsten Hoefler, and Hugh Leather. 2020 · 2003
Earlier work this paper cites.
Software language comprehension using a program-derived semantic graph
Original
Roshni G. Iyer, Yizhou Sun, Wei Wang, and Justin Gottschlich. 2020 · 2004
Earlier work this paper cites.
Context-free grammar induction using genetic programming
Faizan Javed, Barrett R. Bryant, Matej Crepinsek, Marjan Mernik, and Alan P. Sprague. 2004 · 2004
Earlier work this paper cites.
On the generalizability of neural program analyzers with respect to semantic-preserving program transformations
Original
Md. Rafiqul Islam Rabin, Nghi D. Q. Bui, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2020 · 2008
Earlier work this paper cites.
Dynamic inference of static types for ruby
Jong-hoon (David) An, Avik Chaudhuri, Jeffrey S. Foster, and Michael Hicks. 2011 · 2011
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013 · 2013
Earlier work this paper cites.
Summarizing source code using a neural attention model
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016 · 2016
Earlier work this paper cites.
Probabilistic model for code with decision trees
Veselin Raychev, Pavol Bielik, and Martin T. Vechev. 2016 · 2016
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Earlier work this paper cites.
Deep learning code fragments for code clone detection
Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016 · 2016
Earlier work this paper cites.
Semi-supervised classification with graph convolutional networks
Thomas N. Kipf and Max Welling. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018 · 2018
Earlier work this paper cites.
Neural code comprehension: A learnable representation of code semantics
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018 · 2018
Earlier work this paper cites.
Improving automatic source code summarization via deep reinforcement learning
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018 · 2018
Earlier work this paper cites.
Learning to mine aligned code and natural language pairs from stack overflow
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018
Earlier work this paper cites.
code2vec: learning distributed representations of code
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019 · 2019
Earlier work this paper cites.

Similar

BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Graph matching networks for learning the similarity of graph structured objects
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. 2019 · 2019
Cited alongside, same era.
Ithemal: Accurate, portable and fast basic block throughput estimation using deep neural networks
Charith Mendis, Alex Renda, Saman P. Amarasinghe, and Michael Carbin. 2019 · 2019
Cited alongside, same era.
Multi-modal attention network learning for semantic source code retrieval
Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou Zhao, Jian Wu, and Philip S. Yu. 2019 · 2019
Cited alongside, same era.
A simple framework for contrastive learning of visual representations
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020 · 2020
Cited alongside, same era.
ELECTRA: pre-training text encoders as discriminators rather than generators
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020 · 2020
Cited alongside, same era.
Codebert: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020 · 2020
Cited alongside, same era.
Global relational models of source code
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2020 · 2020
Cited alongside, same era.
Learning and evaluating contextual embedding of source code
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020 · 2020
Cited alongside, same era.
BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020
Cited alongside, same era.
Deep learning for software defect prediction: A survey
Safa Omri and Carsten Sinz. 2020 · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.

Then

Graphcodebert: Pre-training code representations with data flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021 · 2021
Later among the works it cites.
Cosqa: 20, 000+ web queries for code search and question answering
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021 · 2021
Later among the works it cites.
Contrastive code representation learning
Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021 · 2021
Later among the works it cites.
Treebert: A tree-based pre-trained model for programming language
Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021 · 2021
Later among the works it cites.
Code prediction by feeding trees to transformers
Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021 · 2021
Later among the works it cites.
Deep graph matching and searching for semantic code retrieval
Xiang Ling, Lingfei Wu, Saizhuo Wang, Gaoning Pan, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, and Shouling Ji. 2021 · 2021
Later among the works it cites.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021 · 2021
Later among the works it cites.
Cotext: Multi-task learning with code-text transformer
Original
Long N. Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James T. Anibal, Alec Peltekian, and Yanfang Ye. 2021 · 2021
Later among the works it cites.
Project codenet: A large-scale AI for code dataset for learning a diversity of coding tasks
Original
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. 2021 · 2021
Later among the works it cites.
Time-aware user modeling with check-in time prediction for next POI recommendation
Xin Wang, Xiao Liu, Li Li, Xiao Chen, Jin Liu, and Hao Wu. 2021a · 2021
Later among the works it cites.
Relational graph neural network with neighbor interactions for bundle recommendation service
Xin Wang, Xiao Liu, Jin Liu, and Hao Wu. 2021b · 2021
Later among the works it cites.
Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. 2021 · 2021
Later among the works it cites.
Servicebert: A pre-trained model for web service tagging and recommendation
Xin Wang, Pingyi Zhou, Yasheng Wang, Xiao Liu, Jin Liu, and Hao Wu. 2021a · 2021
Later among the works it cites.
Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021b · 2021
Later among the works it cites.
Mashup-oriented web api recommendation via multi-model fusion and multi-task learning
Hao Wu, Yunhao Duan, Kun Yue, and Lei Zhang. 2021 · 2021
Later among the works it cites.
Scalpel: The python static analysis framework
Original
Li Li, Jiawei Wang, and Haowei Quan. 2022 · 2022
Closest in time.
Compilable neural code generation with compiler feedback
Original
Xin Wang, Yasheng Wang, Yao Wan, Fei Mi, Yitong Li, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, and Qun Liu. 2022 · 2022
Closest in time.
Exploiting gated graph neural network for detecting and explaining self-admitted technical debts
Jiaojiao Yu, Kunsong Zhao, Jin Liu, Xiao Liu, Zhou Xu, and Xin Wang. 2022 · 2022
Closest in time.
Graph4web: A relation-aware graph attention network for web service classification
Kunsong Zhao, Jin Liu, Zhou Xu, Xiao Liu, Lei Xue, Zhiwen Xie, Yuxuan Zhou, and Xin Wang. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…