Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMs

Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022 · 1965

Earlier work this paper cites.

The information bottleneck method

Naftali Tishby, Fernando C Pereira, and William Bialek. 2000 · 2000

Earlier work this paper cites.

Mining and summarizing customer reviews

Minqing Hu and Bing Liu. 2004 · 2004

Earlier work this paper cites.

A tutorial on the cross-entropy method

Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. 2005 · 2005

Earlier work this paper cites.

Ape210k: A large-scale and template-rich dataset of math word problems

Original

Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020 · 2009

Earlier work this paper cites.

Recursive deep models for semantic compositionality over a sentiment treebank

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013 · 2013

Earlier work this paper cites.

Deep learning and the information bottleneck principle

Naftali Tishby and Noga Zaslavsky. 2015 · 2015

Earlier work this paper cites.

Character-level convolutional networks for text classification

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 · 2015

Earlier work this paper cites.

Badnets: Identifying vulnerabilities in the machine learning model supply chain

Original

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017 · 2017

Earlier work this paper cites.

A backdoor attack against lstm-based text classification systems

Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019 · 2019

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Knowledge distillation for multi-task learning

Wei-Hong Li and Hakan Bilen. 2020 · 2020

Earlier work this paper cites.

Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification

Chuanshuai Chen and Jiazhu Dai. 2021 · 2021

Earlier work this paper cites.

Deep feature space trojan attack of neural networks by controlled detoxification

Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. 2021 · 2021

Earlier work this paper cites.

Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation

Yunjie Ge, Qian Wang, Baolin Zheng, Xinlu Zhuang, Qi Li, Chao Shen, and Cong Wang. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021 · 2021

Earlier work this paper cites.

Fedpara: Low-rank hadamard product for communication-efficient federated learning

Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. 2021 · 2021

Earlier work this paper cites.

Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation

Original

Taehyeon Kim, Jaehoon Oh, NakYil Kim, Sangwook Cho, and Se-Young Yun. 2021 · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Earlier work this paper cites.

Backdoor attacks on pre-trained models by layerwise weight poisoning

Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. 2021a · 2021

Earlier work this paper cites.

Hidden backdoors in human-centric language models

Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu. 2021b · 2021

Earlier work this paper cites.

Prefix-tuning: Optimizing continuous prompts for generation

Xiang Lisa Li and Percy Liang. 2021 · 2021

Earlier work this paper cites.

Onion: A simple and effective defense against textual backdoor attacks

Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a · 2021

Earlier work this paper cites.

Concealed data poisoning attacks on nlp models

Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021 · 2021

Cited alongside, same era.

Badprompt: Backdoor attacks on continuous prompts

Xiangrui Cai, Sihan Xu, Ying Zhang, Xiaojie Yuan, et al. 2022 · 2022

Cited alongside, same era.

Kallima: A clean-label framework for textual backdoor attacks

Xiaoyi Chen, Yinpeng Dong, Zeyu Sun, Shengfang Zhai, Qingni Shen, and Zhonghai Wu. 2022 · 2022

Cited alongside, same era.

Triggerless backdoor attack for nlp tasks with clean labels

Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Yi Yang, Shangwei Guo, and Chun Fan. 2022 · 2022

Cited alongside, same era.

Badhash: Invisible backdoor attacks against deep hashing with clean label

Shengshan Hu, Ziqi Zhou, Yechao Zhang, Leo Yu Zhang, Yifeng Zheng, et al. 2022 · 2022

Cited alongside, same era.

Backdoor attack against nlp models with robustness-aware perturbation defense

Prompt as triggers for backdoor attack: Examining the vulnerability in language models

Shuai Zhao, Jinming Wen, Anh Tuan Luu, Junbo Zhao, and Jie Fu. 2023b · 2023

Later among the works it cites.

Adfl: Defending backdoor attacks in federated learning via adversarial distillation

Chengcheng Zhu, Jiale Zhang, Xiaobing Sun, Bing Chen, and Weizhi Meng. 2023 · 2023

Later among the works it cites.

Llama 3 model card

AI@Meta. 2024 · 2024

Closest in time.

Mitigating backdoor attacks in pre-trained encoders via self-supervised knowledge distillation

Rongfang Bie, Jinxiu Jiang, Hongcheng Xie, Yu Guo, Yinbin Miao, and Xiaohua Jia. 2024 · 2024

Closest in time.

Robust knowledge distillation based on feature variance against backdoored teacher model

Original

Jinyin Chen, Xiaoming Zhao, Haibin Zheng, Xiao Li, Sheng Xiang, and Haifeng Guo. 2024 · 2024

Closest in time.