2023

An Empirical Comparison of Pre-Trained Models of Source Code

Niu, Changan, Li, Chuanyi, Ng, Vincent et al.

Understand

While a large number of pre-trained models of source code have been successfully developed and applied to a variety of software engineering (SE) tasks in recent years, our understanding of these pre-trained models is arguably fairly limited.

With the goal of advancing our understanding of these models, we perform the first systematic empirical comparison of 19 recently-developed pre-trained models of source code on 13 SE tasks.
To gain additional insights into these models, we adopt a recently-developed 4-dimensional categorization of pre-trained models, and subsequently investigate whether there are correlations between different categories of pre-trained models and their performances on different SE tasks.

Built on

E. S. Edgington, “Approximate randomization tests,” The Journal of Psychology , vol. 72, no. 2, pp. 143–149, 1969
1969
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318
2002
Earlier work this paper cites.
C. Cummins, H. Leather, Z. Fisches, T. Ben-Nun, T. Hoefler, and M. O’Boyle, “Deep data flow analysis,” 2020. [Online]. Available: https://arxiv.org/abs/2012.01470
Original
2012
Earlier work this paper cites.
J. Svajlenko, J. F. Islam, I. Keivanloo, C. K. Roy, and M. M. Mia, “Towards a big data curated benchmark of inter-project code clones,” in Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution , 2014, pp. 476–480
2014
Earlier work this paper cites.
A. M. Dai and Q. V. Le, “Semi-supervised sequence learning,” Advances in neural information processing systems , vol. 28, 2015
2015
Earlier work this paper cites.
L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , 2016, pp. 1287–1293
2016
Earlier work this paper cites.
V. Raychev, P. Bielik, and M. Vechev, “Probabilistic model for code with decision trees,” in Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications , 2016, pp. 731–747
2016
Earlier work this paper cites.
A. V. Miceli-Barone and R. Sennrich, “A parallel corpus of python functions and documentation strings for automated code documentation and code generation,” in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , 2017, pp. 314–319
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017
2017
Earlier work this paper cites.
M. D. Ernst, “Natural language is a programming language: Applying natural language processing to software development,” in 2nd Summit on Advances in Programming Languages (SNAPL 2017) . Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2017
2017
Earlier work this paper cites.
J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2018, pp. 328–339
2018
Earlier work this paper cites.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) . New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2227–2237. [Online]. Available: https://aclanthology.org/N18-1202
2018
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
T. Ben-Nun, A. S. Jakobovits, and T. Hoefler, “Neural code comprehension: A learnable representation of code semantics,” Advances in Neural Information Processing Systems , vol. 31, 2018
2018
Earlier work this paper cites.
M. Allamanis, M. Brockschmidt, and M. Khademi, “Learning to represent programs with graphs,” in International Conference on Learning Representations , 2018
2018
Earlier work this paper cites.
M. Pradel and K. Sen, “Deepbugs: A learning approach to name-based bug detection,” Proceedings of the ACM on Programming Languages , vol. 2, no. OOPSLA, pp. 1–25, 2018
2018
Earlier work this paper cites.
X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC) . IEEE, 2018, pp. 200–20 010
2018
Earlier work this paper cites.
X. Hu, G. Li, X. Xia, D. Lo, S. Lu, and Z. Jin, “Summarizing source code with transferred api knowledge,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018 , 2018, pp. 2269–2275
2018
Earlier work this paper cites.
S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Mapping language to code in programmatic context,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018, pp. 1643–1652
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019, pp. 4171–4186
2019
Earlier work this paper cites.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” Advances in neural information processing systems , vol. 32, 2019
2019
Earlier work this paper cites.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
Original
2019
Earlier work this paper cites.
K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” in International Conference on Learning Representations , 2019
2019
Earlier work this paper cites.

Similar

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Cited alongside, same era.
U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: Learning distributed representations of code,” Proceedings of the ACM on Programming Languages , vol. 3, no. POPL, pp. 1–29, 2019
2019
Cited alongside, same era.
J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu, “A novel neural source code representation based on abstract syntax tree,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 783–794
2019
Cited alongside, same era.
Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” Advances in neural information processing systems , vol. 32, 2019
2019
Cited alongside, same era.
K. W. Nafi, T. S. Kar, B. Roy, C. K. Roy, and K. A. Schneider, “Clcdsa: cross language code clone detection using syntactical features and api documentation,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 1026–1037
2019
Cited alongside, same era.
M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural machine translation,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 28, no. 4, pp. 1–29, 2019
2019
Cited alongside, same era.
M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk, “Learning how to mutate source code from bug-fixes,” in 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE Computer Society, 2019, pp. 301–312
2019
Cited alongside, same era.
U. Alon, S. Brody, O. Levy, and E. Yahav, “code2seq: Generating sequences from structured representations of code,” in International Conference on Learning Representations , 2019
2019
Cited alongside, same era.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , vol. 21, pp. 1–67, 2020
2020
Cited alongside, same era.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020, pp. 7871–7880
2020
Cited alongside, same era.
A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in International Conference on Machine Learning . PMLR, 2020, pp. 5110–5121
2020
Cited alongside, same era.
L. Buratti, S. Pujar, M. Bornea, S. McCarley, Y. Zheng, G. Rossiello, A. Morari, J. Laredo, V. Thost, Y. Zhuang et al. , “Exploring software naturalness through neural language models,” arXiv preprint arXiv:2006.12641 , 2020
Original
2020
Cited alongside, same era.

Then

A. Karmakar and R. Robbes, “What do pre-trained code models know about code?” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 1332–1336
2021
Later among the works it cites.
N. D. Bui, Y. Yu, and L. Jiang, “Infercode: Self-supervised learning of code representations by predicting subtrees,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 1186–1197
2021
Later among the works it cites.
X. Jiang, Z. Zheng, C. Lyu, L. Li, and L. Lyu, “Treebert: A tree-based pre-trained model for programming language,” in Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence , vol. 161. PMLR, 27–30 Jul 2021, pp. 54–63
2021
Later among the works it cites.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU, “CodeXGLUE: A machine learning benchmark dataset for code understanding and generation,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021
2021
Later among the works it cites.
J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, and N. Duan, “Cosqa: 20,000+ web queries for code search and question answering,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021, pp. 5690–5700
2021
Later among the works it cites.
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. LIU, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in International Conference on Learning Representations , 2021
2021
Later among the works it cites.
B. Roziere, M.-A. Lachaux, M. Szafraniec, and G. Lample, “Dobf: A deobfuscation pre-training objective for programming languages,” arXiv preprint arXiv:2102.07492 , 2021
Original
2021
Later among the works it cites.
A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshyvanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 336–347
2021
Later among the works it cites.
W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021, pp. 2655–2668
2021
Later among the works it cites.
W. Qi, Y. Gong, Y. Yan, C. Xu, B. Yao, B. Zhou, B. Cheng, D. Jiang, J. Chen, R. Zhang et al. , “Prophetnet-x: Large-scale pre-training models for english, chinese, multi-lingual, dialog, and code generation,” arXiv preprint arXiv:2104.08006 , 2021
Original
2021
Later among the works it cites.
L. Phan, H. Tran, D. Le, H. Nguyen, J. Annibal, A. Peltekian, and Y. Ye, “Cotext: Multi-task learning with code-text transformer,” in Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021) , 2021, pp. 40–47
2021
Later among the works it cites.
D. Peng, S. Zheng, Y. Li, G. Ke, D. He, and T.-Y. Liu, “How could neural networks understand programs?” in International Conference on Machine Learning . PMLR, 2021, pp. 8476–8486
2021
Later among the works it cites.
J. Zhang, H. Hong, Y. Zhang, Y. Wan, Y. Liu, and Y. Sui, “Disentangled code representation learning for multiple programming languages,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , 2021, pp. 4454–4466
2021
Later among the works it cites.
Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021, pp. 8696–8708
2021
Later among the works it cites.
X. Wang, Y. Wang, F. Mi, P. Zhou, Y. Wan, X. Liu, L. Li, H. Wu, J. Liu, and X. Jiang, “Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation,” arXiv preprint arXiv:2108.04556 , 2021
Original
2021
Later among the works it cites.
W. Ma, M. Zhao, E. Soremekun, Q. Hu, J. M. Zhang, M. Papadakis, M. Cordy, X. Xie, and Y. L. Traon, “Graphcode2vec: generic code embedding via lexical and program dependence analyses,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 524–536
2022
Later among the works it cites.
K. Zhang, W. Wang, H. Zhang, G. Li, and Z. Jin, “Learning to represent programs with heterogeneous graphs,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension , 2022, pp. 378–389
2022
Later among the works it cites.
C. Niu, C. Li, B. Luo, and V. Ng, “Deep learning meets software engineering: A survey on pre-trained models of source code,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022 , 2022, pp. 5546–5555
2022
Later among the works it cites.
C. Niu, C. Li, V. Ng, J. Ge, L. Huang, and B. Luo, “Spt-code: Sequence-to-sequence pre-training for learning source code representations,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) , 2022, pp. 01–13
2022
Later among the works it cites.
D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022, pp. 7212–7225
2022
Later among the works it cites.
Z. Zeng, H. Tan, H. Zhang, J. Li, Y. Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2022, pp. 39–51
2022
Later among the works it cites.
D. Wang, Z. Jia, S. Li, Y. Yu, Y. Xiong, W. Dong, and X. Liao, “Bridging pre-trained models and downstream tasks for source code understanding,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 287–298
2022
Later among the works it cites.
Z. Zhang, H. Zhang, B. Shen, and X. Gu, “Diet code is healthy: Simplifying programs for pre-trained models of code,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1073–1084
2022
Later among the works it cites.
J. Shi, Z. Yang, B. Xu, H. J. Kang, and D. Lo, “Compressing pre-trained models of code into 3 mb,” in The 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022 , 2022
Original
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…