2023

Keeping Pace with Ever-Increasing Data: Towards Continual Learning of Code Intelligence Models

Gao, Shuzheng, Zhang, Hongyu, Gao, Cuiyun et al.

Understand

Previous research on code intelligence usually trains a deep learning model on a fixed dataset in an offline manner.

However, in real-world scenarios, new code repositories emerge incessantly, and the carried new knowledge is beneficial for providing up-to-date code intelligence services to developers.
In this paper, we aim at the following problem: How to enable code intelligence models to continually learn from ever-increasing data? One major challenge here is catastrophic forgetting, meaning that the model can easily forget knowledge learned from previous datasets when learning from the new dataset.
To tackle this challenge, we propose REPEAT, a novel method for continual learning of code intelligence models.

Built on

M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation . Elsevier, 1989, vol. 24, pp. 109–165
1989
Earlier work this paper cites.
T. Kamiya, S. Kusumoto, and K. Inoue, “Ccfinder: A multilinguistic token-based code clone detection system for large scale source code,” IEEE Trans. Software Eng. , vol. 28, no. 7, pp. 654–670, 2002
2002
Earlier work this paper cites.
K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA . ACL, 2002, pp. 311–318
2002
Earlier work this paper cites.
J. Ramos et al. , “Using tf-idf to determine word relevance in document queries,” in Proceedings of the first instructional conference on machine learning , vol. 242, no. 1. Citeseer, 2003, pp. 29–48
2003
Earlier work this paper cites.
C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81
2004
Earlier work this paper cites.
M. Kim, V. Sazawal, D. Notkin, and G. C. Murphy, “An empirical study of code clone genealogies,” in Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon, Portugal, September 5-9, 2005 , M. Wermelinger and H. C. Gall, Eds. ACM, 2005, pp. 187–196
2005
Earlier work this paper cites.
S. Banerjee and A. Lavie, “METEOR: an automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005 , J. Goldstein, A. Lavie, C. Lin, and C. R. Voss, Eds. Association for Computational Linguistics, 2005, pp. 65–72
2005
Earlier work this paper cites.
S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 . The Association for Computer Linguistics, 2016
2016
Earlier work this paper cites.
L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence . AAAI Press, 2016, pp. 1287–1293
2016
Earlier work this paper cites.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3526, 2017
2017
Earlier work this paper cites.
H. Wei and M. Li, “Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017 , C. Sierra, Ed. ijcai.org, 2017, pp. 3034–3040
2017
Earlier work this paper cites.
D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 6467–6476
2017
Earlier work this paper cites.
R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 . IEEE Computer Society, 2017, pp. 7120–7129
2017
Earlier work this paper cites.
X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in Proceedings of the 26th Conference on Program Comprehension, ICPC 2018, Gothenburg, Sweden, May 27-28, 2018 , F. Khomh, C. K. Roy, and J. Siegmund, Eds. ACM, 2018, pp. 200–210
2018
Earlier work this paper cites.
X. Gu, H. Zhang, and S. Kim, “Deep code search,” in Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018 , M. Chaudron, I. Crnkovic, M. Chechik, and M. Harman, Eds. ACM, 2018, pp. 933–944
2018
Earlier work this paper cites.
Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in 25th Annual Network and Distributed System Security Symposium, NDSS 2018 . The Internet Society, 2018
2018
Earlier work this paper cites.
Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 40, no. 12, pp. 2935–2947, 2018
2018
Earlier work this paper cites.
Z. Chen and B. Liu, “Lifelong machine learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning , vol. 12, no. 3, pp. 1–207, 2018
2018
Earlier work this paper cites.
A. Mallya and S. Lazebnik, “Packnet: Adding multiple tasks to a single network by iterative pruning,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 . Computer Vision Foundation / IEEE Computer Society, 2018, pp. 7765–7773
2018
Earlier work this paper cites.
R. Kemker and C. Kanan, “Fearnet: Brain-inspired model for incremental learning,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018
2018
Earlier work this paper cites.
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. W. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , 2018, pp. 8536–8546
2018
Earlier work this paper cites.

Similar

Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural-machine-translation-based commit message generation: how far are we?” in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 , M. Huchard, C. Kästner, and G. Fraser, Eds. ACM, 2018, pp. 373–384
2018
Cited alongside, same era.
Y. Zhou, S. Liu, J. K. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019 , 2019, pp. 10 197–10 207
2019
Cited alongside, same era.
H. Wang, W. Xiong, M. Yu, X. Guo, S. Chang, and W. Y. Wang, “Sentence embedding alignment for lifelong relation extraction,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, 2019, pp. 796–806
2019
Cited alongside, same era.
M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” in Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, Athens, Greece, October 23-24, 2019 , H. Masuhara and T. Petricek, Eds. ACM, 2019, pp. 143–153
2019
Cited alongside, same era.
H. Yu, W. Lam, L. Chen, G. Li, T. Xie, and Q. Wang, “Neural detection of semantic code clones via tree-based convolution,” in Proceedings of the 27th International Conference on Program Comprehension, ICPC 2019, Montreal, QC, Canada, May 25-31, 2019 , Y. Guéhéneuc, F. Khomh, and F. Sarro, Eds. IEEE / ACM, 2019, pp. 70–80
2019
Cited alongside, same era.
T. Ramalho and M. Garnelo, “Adaptive posterior learning: few-shot learning with a surprise-based memory module,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019
2019
Cited alongside, same era.
A. LeClair and C. McMillan, “Recommendations for datasets for source code summarization,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, 2019, pp. 3931–3937
2019
Cited alongside, same era.
J. Huang, L. Qu, R. Jia, and B. Zhao, “O2u-net: A simple noisy label detection approach for deep neural networks,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 . IEEE, 2019, pp. 3325–3333
2019
Cited alongside, same era.
H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” CoRR , vol. abs/1909.09436, 2019
Original
2019
Cited alongside, same era.
J. Cambronero, H. Li, S. Kim, K. Sen, and S. Chandra, “When deep learning met code search,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Estonia, August 26-30, 2019 , M. Dumas, D. Pfahl, S. Apel, and A. Russo, Eds. ACM, 2019, pp. 964–974
2019
Cited alongside, same era.
W. U. Ahmad, S. Chakraborty, B. Ray, and K. Chang, “A transformer-based approach for source code summarization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 . Association for Computational Linguistics, 2020, pp. 4998–5007
2020
Cited alongside, same era.
W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural networkand flow-augmented abstract syntax tree,” CoRR , vol. abs/2002.08653, 2020
Original
2020
Cited alongside, same era.

Then

D. Zou, Y. Zhu, S. Xu, Z. Li, H. Jin, and H. Ye, “Interpreting deep learning-based vulnerability detector predictions based on heuristic searching,” ACM Trans. Softw. Eng. Methodol. , vol. 30, no. 2, pp. 23:1–23:31, 2021
2021
Later among the works it cites.
Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 . Association for Computational Linguistics, 2021, pp. 8696–8708
2021
Later among the works it cites.
H. Wu, H. Zhao, and M. Zhang, “Code summarization with structure-induced transformer,” in Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021 , ser. Findings of ACL, C. Zong, F. Xia, W. Li, and R. Navigli, Eds., vol. ACL/IJCNLP 2021. Association for Computational Linguistics, 2021, pp. 1078–1090
2021
Later among the works it cites.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” CoRR , vol. abs/2102.04664, 2021
Original
2021
Later among the works it cites.
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou, “Graphcodebert: Pre-training code representations with data flow,” in 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021
2021
Later among the works it cites.
Z. Li, L. Qu, and G. Haffari, “Total recall: a customized continual learning method for neural semantic parsers,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , M. Moens, X. Huang, L. Specia, and S. W. Yih, Eds. Association for Computational Linguistics, 2021, pp. 3816–3831
2021
Later among the works it cites.
S. Liu, Y. Chen, X. Xie, J. K. Siow, and Y. Liu, “Retrieval-augmented generation for code summarization via hybrid GNN,” in 9th International Conference on Learning Representations, ICLR 2021 . OpenReview.net, 2021
2021
Later among the works it cites.
W. Gu, Z. Li, C. Gao, C. Wang, H. Zhang, Z. Xu, and M. R. Lyu, “Cradle: Deep code retrieval based on semantic dependency learning,” Neural Networks , vol. 141, pp. 385–394, 2021
2021
Later among the works it cites.
C. Wang, Y. Yang, C. Gao, Y. Peng, H. Zhang, and M. R. Lyu, “No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022 , A. Roychoudhury, C. Cadar, and M. Kim, Eds. ACM, 2022, pp. 382–394
2022
Later among the works it cites.
D. Zan, B. Chen, D. Yang, Z. Lin, M. Kim, B. Guan, Y. Wang, W. Chen, and J. Lou, “CERT: continual pre-training on sketches for library-oriented code generation,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , L. D. Raedt, Ed. ijcai.org, 2022, pp. 2369–2375
2022
Later among the works it cites.
H. Yu, X. Hu, G. Li, Y. Li, Q. Wang, and T. Xie, “Assessing and improving an evaluation dataset for detecting semantic code clones via deep learning,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 4, 2022
2022
Later among the works it cites.
M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 7, pp. 3366–3385, 2022
2022
Later among the works it cites.
W. Yuan, H. Yin, T. He, T. Chen, Q. Wang, and L. Cui, “Unified question generation with continual lifelong learning,” in WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022 , F. Laforest, R. Troncy, E. Simperl, D. Agarwal, A. Gionis, I. Herman, and L. Médini, Eds. ACM, 2022, pp. 871–881
2022
Later among the works it cites.
Z. Sun, L. Li, Y. Liu, X. Du, and L. Li, “On the importance of building high-quality training datasets for neural code search,” in 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . ACM, 2022, pp. 1609–1620
2022
Later among the works it cites.
L. Shi, F. Mu, X. Chen, S. Wang, J. Wang, Y. Yang, G. Li, X. Xia, and Q. Wang, “Are we building on the rock? on the importance of data preprocessing for code summarization,” CoRR , vol. abs/2207.05579, 2022
Original
2022
Later among the works it cites.
Z. Tang, X. Shen, C. Li, J. Ge, L. Huang, Z. Zhu, and B. Luo, “Ast-trans: Code summarization with efficient tree-structured attention,” in 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022 . IEEE, 2022, pp. 150–162
2022
Later among the works it cites.
S. Gao, C. Gao, Y. He, J. Zeng, L. Y. Nie, X. Xia, and M. R. Lyu, “Code structure guided transformer for source code summarization,” ACM Trans. Softw. Eng. Methodol. , 2022
2022
Later among the works it cites.
X. Wen, C. Gao, J. Ye, Z. Tian, Y. Jia, and X. Wang, “Meta-path based attentional graph learning model for vulnerability detection,” CoRR , vol. abs/2212.14274, 2022
Original
2022
Later among the works it cites.
W. Yuan, Q. Zhang, T. He, C. Fang, N. Q. V. Hung, X. Hao, and H. Yin, “CIRCLE: continual repair across programming languages,” in ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022 , S. Ryu and Y. Smaragdakis, Eds. ACM, 2022, pp. 678–690
2022
Later among the works it cites.
S. Liu, C. Gao, S. Chen, L. Y. Nie, and Y. Liu, “ATOM: commit message generation based on abstract syntax tree and hybrid ranking,” IEEE Trans. Software Eng. , vol. 48, no. 5, pp. 1800–1817, 2022
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…