Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey

S. Sivanandam, S. Deepa, S. Sivanandam, and S. Deepa, Genetic algorithms . Springer, 2008

2008

Earlier work this paper cites.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Proceedings of the European Conference on Computer Vision , 2014

2014

Earlier work this paper cites.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Proceedings of the European Conference on Computer Vision , 2014

2014

Earlier work this paper cites.

B. Saleh and A. Elgammal, “Large-scale classification of fine-art paintings: Learning the right metric on the right feature,” arXiv preprint arXiv:1505.00855 , 2015

Original

2015

Earlier work this paper cites.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in Proceedings of the Advances in Neural Information Processing Systems , 2017

2017

Earlier work this paper cites.

P. Bedapudi, “Nudenet: Neural nets for nudity classification, detection and selective censoring,” 2019

2019

Earlier work this paper cites.

“Image censorship,” https://github.com/lucasxlu/XCloud/ tree/master/research/imgcensor, 2019

2019

Earlier work this paper cites.

Y. Liu, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

Original

2019

Earlier work this paper cites.

J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Proceedings of the Advances in Neural Information Processing Systems , 2020

2020

Earlier work this paper cites.

L. Hanu and Unitary team, “Detoxify,” Github. https://github.com/unitaryai/detoxify, 2020

2020

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learning , 2021

2021

Earlier work this paper cites.

M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision , 2021

2021

Earlier work this paper cites.

J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y. Choi, “Clipscore: A reference-free evaluation metric for image captioning,” in Proceedings of the Empirical Methods in Natural Language Processing , 2021

2021

Earlier work this paper cites.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022

2022

Earlier work this paper cites.

J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” in Proceedings of the Advances in Neural Information Processing Systems , 2022

2022

Earlier work this paper cites.

J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598 , 2022

Original

2022

Earlier work this paper cites.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al. , “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022

Original

2022

Earlier work this paper cites.

J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tramèr, “Red-teaming the stable diffusion safety filter,” in Proceedings of the Advances in Neural Information Processing Systems Workshops , 2022

2022

Earlier work this paper cites.

P. Schramowski, C. Tauchmann, and K. Kersting, “Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?” in Proceedings of the ACM Conference on Fairness, Accountability, and Transparency , 2022

2022

Earlier work this paper cites.

A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” in Proceedings of the Findings of the Association for Computational Linguistics , 2022

2022

Earlier work this paper cites.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” in Proceedings of the Empirical Methods in Natural Language Processing , 2023

2023

Earlier work this paper cites.

J. Christian, “Amazing “jailbreak” bypasses chatgpt’s ethics safeguards,” Futurism, February , 2023

2023

Earlier work this paper cites.

Albert, “Jailbreak chat,” https://www.jailbreakchat.com/ , 2023

2023

Earlier work this paper cites.

H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” in Proceedings of the Findings of Empirical Methods in Natural Language Processing , 2023

2023

Earlier work this paper cites.

A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. C. H. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122 , 2023

Original

2023

Earlier work this paper cites.

P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov et al. , “Audiopalm: A large language model that can speak and listen,” arXiv preprint arXiv:2306.12925 , 2023

Original

2023

Earlier work this paper cites.

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 500–22 510

2023

Earlier work this paper cites.

J. Xu, X. Wang, Y.-P. Cao, W. Cheng, Y. Shan, and S. Gao, “Instructp2p: Learning to edit 3d point clouds with text instructions,” arXiv preprint arXiv:2306.07154 , 2023

Original

2023

Earlier work this paper cites.

A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts et al. , “Stable video diffusion: Scaling latent video diffusion models to large datasets,” arXiv preprint arXiv:2311.15127 , 2023

Original

2023

Earlier work this paper cites.

P. Lu, B. Peng, H. Cheng, M. Galley, K. Chang, Y. N. Wu, S. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al. , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” https://vicuna. lmsys. org , 2023

2023

Earlier work this paper cites.

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

Original

2023

Earlier work this paper cites.

Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860 , 2023

Original

2023

Earlier work this paper cites.

A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043 , 2023

Original

2023

Earlier work this paper cites.

Y. Deng and H. Chen, “Divide-and-conquer attack: Harnessing the power of llm to bypass the censorship of text-to-image generation model,” arXiv preprint arXiv:2312.07130 , 2023

Original

2023

Earlier work this paper cites.

Y. Gong, D. Ran, J. Liu, C. Wang, T. Cong, A. Wang, S. Duan, and X. Wang, “Figstep: Jailbreaking large vision-language models via typographic visual prompts,” arXiv preprint arXiv:2311.05608 , 2023

Original

2023

Earlier work this paper cites.

Y. Wu, X. Li, Y. Liu, P. Zhou, and L. Sun, “Jailbreaking gpt-4v via self-adversarial attacks with system prompts,” arXiv preprint arXiv:2311.09127 , 2023

Original

2023

Earlier work this paper cites.

H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, “How many unicorns are in this image? a safety evaluation benchmark for vision llms,” arXiv preprint arXiv:2311.16101 , 2023

Original

2023

Earlier work this paper cites.

N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt, “Are aligned neural networks adversarially aligned?” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision-language models,” in Proceedings of the Advances in Neural Information Processing Systems , 2023

2023

Earlier work this paper cites.

“Leonardo.Ai,” https://leonardo.ai/2023

2023

Earlier work this paper cites.

X. Zhang, C. Zhang, T. Li, Y. Huang, X. Jia, X. Xie, Y. Liu, and C. Shen, “Jailguard: A universal detection framework for llm prompt-based attacks,” arXiv preprint arXiv:2312.10766 , 2023

Original

2023

Earlier work this paper cites.

R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau, “Erasing concepts from diffusion models,” in Proceedings of the IEEE International Conference on Computer Vision , 2023

2023

Earlier work this paper cites.

C.-P. Huang, K.-P. Chang, C.-T. Tsai, Y.-H. Lai, and Y.-C. F. Wang, “Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers,” in Proceedings of the European Conference on Computer Vision , 2023

2023

Earlier work this paper cites.

S. Kim, S. Jung, B. Kim, M. Choi, J. Shin, and J. Lee, “Towards safe self-distillation of internet-scale text-to-image diffusion models,” in Proceedings of the International Conference on Machine Learning Workshops , 2023

2023

Earlier work this paper cites.

P. Schramowski, M. Brack, B. Deiseroth, and K. Kersting, “Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023

2023

Earlier work this paper cites.

H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine et al. , “Llama guard: Llm-based input-output safeguard for human-ai conversations,” arXiv preprint arXiv:2312.06674 , 2023

Original

2023

Earlier work this paper cites.

Y. Qu, X. Shen, X. He, M. Backes, S. Zannettou, and Y. Zhang, “Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security , 2023

2023

Earlier work this paper cites.

OpenAI, “Openai usage policy,” 2023, accessed on 10-2023. [Online]. Available: https://openai.com/policies/usage-policies

2023

Earlier work this paper cites.

Meta, “Llama usage policy,” 2023, accessed on 10-2023. [Online]. Available: https://ai.meta.com/llama/use-policy

2023

Earlier work this paper cites.

J. Yu, X. Lin, Z. Yu, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253 , 2023

Original

2023

Earlier work this paper cites.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

Original

2023

Earlier work this paper cites.

H. Liu, Y. Wu, S. Zhai, B. Yuan, and N. Zhang, “Riatig: Reliable and imperceptible adversarial text-to-image generation with natural prompts,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023

2023

Earlier work this paper cites.

C. Team, “Chameleon: Mixed-modal early-fusion foundation models,” arXiv preprint arXiv:2405.09818 , 2024

Original

2024

Earlier work this paper cites.

X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu et al. , “Emu3: Next-token prediction is all you need,” arXiv preprint arXiv:2409.18869 , 2024

Original

2024

Cited alongside, same era.

Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities,” in Proceedings of the International Conference on Machine Learning , 2024

2024

Cited alongside, same era.

J. Xiao, A. Yao, Y. Li, and T.-S. Chua, “Can i trust your answer? visually grounded video question answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2024

2024

Cited alongside, same era.

X. Liu, P. P. Li, H. Huang, Z. Li, X. Cui, W. Deng, Z. He et al. , “Fka-owl: Advancing multimodal fake news detection through knowledge-augmented lvlms,” in Proceedings of the ACM International Conference on Multimedia , 2024

2024

Cited alongside, same era.

Z. Niu, Y. Sun, H. Ren, H. Ji, Q. Wang, X. Ma, G. Hua, and R. Jin, “Efficient llm-jailbreaking by introducing visual modality,” arXiv preprint arXiv:2405.20015 , 2024

Original

2024

Closest in time.

Z. Niu, H. Ren, X. Gao, G. Hua, and R. Jin, “Jailbreaking attack against multimodal large language model,” arXiv preprint arXiv:2402.02309 , 2024

Original

2024

Closest in time.

Y. Li, H. Guo, K. Zhou, W. X. Zhao, and J.-R. Wen, “Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models,” in Proceedings of the European Conference on Computer Vision , 2024

2024

Closest in time.

X. Gu, X. Zheng, T. Pang, C. Du, Q. Liu, Y. Wang, J. Jiang, and M. Lin, “Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast,” in Proceedings of the International Conference on Machine Learning , 2024

2024

Closest in time.