Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation

X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” 2017. [Online]. Available: https://arxiv.org/abs/1712.05526

Original

2017

Earlier work this paper cites.

Y. Ji, X. Zhang, and T. Wang, “Backdoor attacks against learning systems,” in Proc. 2017 IEEE Conf. Commun. Network Secur. (CNS) . IEEE, 2017, pp. 1–9

2017

Earlier work this paper cites.

X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backdoor attacks on deep learning systems using data poisoning,” arXiv preprint arXiv:1712.05526 , 2017

Original

2017

Earlier work this paper cites.

Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 . IEEE Computer Society, 2017, pp. 6325–6334. [Online]. Available: https://doi.org/10.1109/CVPR.2017.670

2017

Earlier work this paper cites.

A. Poplawski, “Self-consistency of voting implies majority vote,” 2018. [Online]. Available: https://arxiv.org/abs/1807.00170

Original

2018

Earlier work this paper cites.

T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” 2019. [Online]. Available: https://arxiv.org/abs/1708.06733

Original

2019

Earlier work this paper cites.

J. Dai, C. Chen, and Y. Li, “A backdoor attack against lstm-based text classification systems,” IEEE Access , vol. 7, pp. 138 872–138 878, 2019

2019

Earlier work this paper cites.

D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 6693–6702

2019

Earlier work this paper cites.

K. Kurita, P. Michel, and G. Neubig, “Weight poisoning attacks on pretrained models,” in Proc. 58th Annu. Meet. Assoc. Comput. Linguist. (ACL) . Association for Computational Linguistics, Jul. 2020, pp. 2793–2806. [Online]. Available: https://aclanthology.org/2020.acl-main.249

2020

Earlier work this paper cites.

X. Chen, A. Salem, D. Chen, M. Backes, S. Ma, Q. Shen, Z. Wu, and Y. Zhang, “Badnl: Backdoor attacks against NLP models with semantic-preserving improvements,” in ACSAC ’21: Annual Computer Security Applications Conference, Virtual Event, USA, December 6 - 10, 2021 . ACM, 2021, pp. 554–569. [Online]. Available: https://doi.org/10.1145/3485832.3485837

2021

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [Online]. Available: https://arxiv.org/abs/2103.00020

Original

2021

Earlier work this paper cites.

F. Qi, M. Li, Y. Chen, Z. Zhang, Z. Liu, Y. Wang, and M. Sun, “Hidden killer: Invisible textual backdoor attacks with syntactic trigger,” in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. (ACL) and 11th Int. Joint Conf. Natural Lang. Process. Online: Association for Computational Linguistics, Aug. 2021, pp. 443–453. [Online]. Available: https://aclanthology.org/2021.acl-long.37

2021

Earlier work this paper cites.

F. Qi, Y. Chen, X. Zhang, M. Li, Z. Liu, and M. Sun, “Mind the style of text! adversarial and backdoor attacks based on text style transfer,” in Proc. 2021 Conf. Empirical Methods Nat. Lang. Process. (EMNLP) . Association for Computational Linguistics, Nov. 2021, pp. 4569–4580. [Online]. Available: https://aclanthology.org/2021.emnlp-main.374

2021

Earlier work this paper cites.

L. Shen, S. Ji, X. Zhang et al. , “Backdoor pre-trained models can transfer to all,” arXiv preprint arXiv:2111.00197 , 2021

Original

2021

Earlier work this paper cites.

L. Shen, S. Ji, X. Zhang, J. Li, J. Chen, J. Shi, C. Fang, J. Yin, and T. Wang, “Backdoor pre-trained models can transfer to all,” in Proc. 2021 ACM SIGSAC Conf. Comput. Commun. Secur. , ser. CCS ’21, 2021, p. 3141–3158. [Online]. Available: https://doi.org/10.1145/3460120.3485370

2021

Earlier work this paper cites.

Y. Li, Y. Li, B. Wu, L. Li, R. He, and S. Lyu, “Invisible backdoor attack with sample-specific triggers,” in 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 16 443–16 452. [Online]. Available: https://doi.org/10.1109/ICCV48922.2021.01615

2021

Earlier work this paper cites.

F. Qi, Y. Chen, X. Zhang, M. Li, Z. Liu, and M. Sun, “Mind the style of text! adversarial and backdoor attacks based on text style transfer,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , M. Moens, X. Huang, L. Specia, and S. W. Yih, Eds. Association for Computational Linguistics, 2021, pp. 4569–4580. [Online]. Available: https://doi.org/10.18653/v1/2021.emnlp-main.374

2021

Earlier work this paper cites.

N. Carlini and A. Terzis, “Poisoning and backdooring contrastive learning,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. [Online]. Available: https://openreview.net/forum?id=iC4UHbQ01Mp

2022

Earlier work this paper cites.

G. Tao, G. Shen, Y. Liu, S. An, Q. Xu, S. Ma, P. Li, and X. Zhang, “Better trigger inversion optimization in backdoor scanning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 13 368–13 378

2022

Earlier work this paper cites.

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id=nZeVKeeFYf9

2022

Earlier work this paper cites.

Z. Sha, X. He, P. Berrang, M. Humbert, and Y. Zhang, “Fine-tuning is all you need to mitigate backdoor attacks,” 2022. [Online]. Available: https://arxiv.org/abs/2212.09067

Original

2022

Earlier work this paper cites.

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, “Palm-e: An embodied multimodal language model,” 2023. [Online]. Available: https://arxiv.org/abs/2303.03378

Original

2023

Earlier work this paper cites.

Z. Yang, X. He, Z. Li, M. Backes, M. Humbert, P. Berrang, and Y. Zhang, “Data poisoning attacks against multimodal encoders,” 2023. [Online]. Available: https://arxiv.org/abs/2209.15266

Original

2023

Earlier work this paper cites.

A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in Proc. Int. Conf. Mach. Learn. (ICML) . PMLR, 2023, pp. 35 413–35 425

2023

Cited alongside, same era.

J. Xu, M. D. Ma, F. Wang, C. Xiao, and M. Chen, “Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models,” arXiv preprint arXiv:2305.14710 , 2023

Original

2023

Cited alongside, same era.

J. Yan, V. Yadav, S. Li, L. Chen, Z. Tang, H. Wang, V. Srinivasan, X. Ren, and H. Jin, “Backdooring instruction-tuned large language models with virtual prompt injection,” in NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly , 2023

2023

Cited alongside, same era.

H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” pp. 26 286–26 296, 2024. [Online]. Available: https://doi.org/10.1109/CVPR52733.2024.02484

2024

Cited alongside, same era.

Y. Li, H. Huang, Y. Zhao, X. Ma, and J. Sun, “Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models,” ArXiv , vol. abs/2408.12798, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271947035

Original

2024

Later among the works it cites.

S. Liang, J. Liang, T. Pang, C. Du, A. Liu, E.-C. Chang, and X. Cao, “Revisiting backdoor attacks against large vision-language models from domain shift,” 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:270764501

2024

Later among the works it cites.

G. B. M. Stan, E. Aflalo, R. Y. Rohekar, A. Bhiwandiwalla, S.-Y. Tseng, M. L. Olson, Y. Gurwicz, C. Wu, N. Duan, and V. Lal, “Lvlm-interpret: An interpretability tool for large vision-language models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.03118

Original

2024

Later among the works it cites.

W. Lyu, J. Yao, S. Gupta, L. Pang, T. Sun, L. Yi, L. Hu, H. Ling, and C. Chen, “Backdooring vision-language models with out-of-distribution data,” in The Thirteenth International Conference on Learning Representations , 2025. [Online]. Available: https://openreview.net/forum?id=tZozeR3VV7