2024

Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: An Empirical Study

Liu, Shuo, Keung, Jacky, Yang, Zhen et al.

Understand

Compared to Full-Model Fine-Tuning (FMFT), Parameter Efficient Fine-Tuning (PEFT) has demonstrated superior performance and lower computational overhead in several code understanding tasks, such as code summarization and code search.

This advantage can be attributed to PEFT's ability to alleviate the catastrophic forgetting issue of Pre-trained Language Models (PLMs) by updating only a small number of parameters.
As a result, PEFT effectively harnesses the pre-trained general-purpose knowledge for downstream tasks.
However, existing studies primarily involve static code comprehension, aligning with the pre-training paradigm of recent PLMs and facilitating knowledge transfer, but they do not account for dynamic code changes.

Built on

Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
Satanjeev Banerjee and Alon Lavie · 2005
Earlier work this paper cites.
What is the long-term impact of changes?
Irina Ioana Brudaru and Andreas Zeller · 2008
Earlier work this paper cites.
A large-scale empirical study of just-in-time quality assurance
Yasutaka Kamei, Emad Shihab, Bram Adams, Ahmed E Hassan, Audris Mockus, Anand Sinha, and Naoyasu Ubayashi · 2012
Earlier work this paper cites.
Boa: A language and infrastructure for analyzing ultra-large-scale software repositories
Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N Nguyen · 2013
Earlier work this paper cites.
On automatically generating commit messages via summarization of source code changes
Luis Fernando Cortés-Coy, Mario Linares-Vásquez, Jairo Aponte, and Denys Poshyvanyk · 2014
Earlier work this paper cites.
Automatically generating commit messages from diffs using neural machine translation
Siyuan Jiang, Ameer Armaly, and Collin McMillan · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
50k-c: a dataset of compilable, and compiled, java projects
Pedro Martins, Rohan Achar, and Cristina Lopes · 2018
Earlier work this paper cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Deepjit: an end-to-end deep learning framework for just-in-time defect prediction
Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi · 2019
Earlier work this paper cites.
Patchnet: Hierarchical deep learning-based stable patch identification for the linux kernel
Thong Hoang, Julia Lawall, Yuan Tian, Richard J Oentaryo, and David Lo · 2019
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly · 2019
Earlier work this paper cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Earlier work this paper cites.
BERT rediscovers the classical NLP pipeline
Ian Tenney, Dipanjan Das, and Ellie Pavlick · 2019
Earlier work this paper cites.
What do you learn from context? probing for sentence structure in contextualized word representations
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick · 2019
Earlier work this paper cites.
Commit message generation for source code changes
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu · 2019
Earlier work this paper cites.
CodeBERT: A pre-trained model for programming and natural languages
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou · 2020
Earlier work this paper cites.
Cc2vec: Distributed representations of code changes
Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall · 2020
Earlier work this paper cites.
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer · 2020
Earlier work this paper cites.
AdapterHub: A framework for adapting transformers
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych · 2020
Earlier work this paper cites.

Similar

Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Cited alongside, same era.
Intellicode compose: Code generation using transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan · 2020
Cited alongside, same era.
Unified pre-training for program understanding and generation
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang · 2021
Cited alongside, same era.
Graphcodebert: Pre-training code representations with data flow
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou · 2021
Cited alongside, same era.
On the effectiveness of adapter-based tuning for pretrained language model adaptation
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si · 2021
Cited alongside, same era.
What do pre-trained code models know about code?
Anjan Karmakar and Romain Robbes · 2021
Cited alongside, same era.
The power of scale for parameter-efficient prompt tuning
Brian Lester, Rami Al-Rfou, and Noah Constant · 2021
Cited alongside, same era.
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang · 2021
Cited alongside, same era.
Codexglue: A machine learning benchmark dataset for code understanding and generation
Original
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu · 2021
Cited alongside, same era.
Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction
Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn · 2021
Cited alongside, same era.
On the evaluation of commit message generation models: An experimental study
Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang · 2021
Cited alongside, same era.
CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi · 2021
Cited alongside, same era.

Then

On the importance of building high-quality training datasets for neural code search
Zhensu Sun, Li Li, Yan Liu, Xiaoning Du, and Li Li · 2022
Later among the works it cites.
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu · 2022
Later among the works it cites.
A survey on cross-lingual summarization
Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou · 2022
Later among the works it cites.
A survey on pretrained language models for neural code intelligence
Original
Yichen Xu and Yanqiao Zhu · 2022
Later among the works it cites.
An extensive study on pre-trained models for program understanding and generation
Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang · 2022
Later among the works it cites.
Differentiable prompt makes pre-trained language models better few-shot learners
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen · 2022
Later among the works it cites.
Xlcost: A benchmark dataset for cross-lingual code intelligence
Original
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy · 2022
Later among the works it cites.
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al · 2023
Later among the works it cites.
On the effectiveness of parameter-efficient fine-tuning
Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier · 2023
Later among the works it cites.
Large language models are few-shot testers: Exploring llm-based general bug reproduction
Sungmin Kang, Juyeon Yoon, and Shin Yoo · 2023
Later among the works it cites.
Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models
Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen · 2023
Later among the works it cites.
Scaling down to scale up: A guide to parameter-efficient fine-tuning
Original
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky · 2023
Later among the works it cites.
Cct5: A code-change-oriented pre-trained model
Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao · 2023
Later among the works it cites.
Ccrep: Learning code change representations via pre-trained code model and query back
Zhongxin Liu, Zhijie Tang, Xin Xia, and Xiaohu Yang · 2023
Later among the works it cites.
Codegen: An open large language model for code with multi-turn program synthesis
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong · 2023
Later among the works it cites.
An empirical comparison of pre-trained models of source code
C. Niu, C. Li, V. Ng, D. Chen, J. Ge, and B. Luo · 2023
Later among the works it cites.
Code llama: Open foundation models for code
Original
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al · 2023
Later among the works it cites.
Utilization of pre-trained language model for adapter-based knowledge transfer in software engineering
Original
Iman Saberi, Fatemeh Fard, and Fuxiang Chen · 2023
Later among the works it cites.
One adapter for all programming languages? adapter tuning for code search and summarization
D. Wang, B. Chen, S. Li, W. Luo, S. Peng, W. Dong, and X. Liao · 2023
Later among the works it cites.
Automated program repair in the era of large pre-trained language models
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang · 2023
Later among the works it cites.
On the significance of category prediction for code-comment synchronization
Zhen Yang, Jacky Wai Keung, Xiao Yu, Yan Xiao, Zhi Jin, and Jingyu Zhang · 2023
Later among the works it cites.
Improving domain-specific neural code generation with few-shot meta-learning
Zhen Yang, Jacky Wai Keung, Zeyu Sun, Yunfei Zhao, Ge Li, Zhi Jin, Shuo Liu, and Yishu Li · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…