Fetching the paper…

Reading the bibliography…

2023

SneakyPrompt: Jailbreaking Text-to-image Generative Models

Yang, Yuchen, Hui, Bo, Yuan, Haolin et al.

Original
PDF
Post on X

Understand

Text-to-image generative models such as Stable Diffusion and DALL$\cdot$E raise many ethical concerns due to the generation of harmful images such as Not-Safe-for-Work (NSFW) ones.

To address these ethical concerns, safety filters are often adopted to prevent the generation of NSFW images.
In this work, we propose SneakyPrompt, the first automated attack framework, to jailbreak text-to-image generative models such that they generate NSFW images even if safety filters are adopted.
Given a prompt that is blocked by a safety filter, SneakyPrompt repeatedly queries the text-to-image generative model and strategically perturbs tokens in the prompt based on the query results to bypass the safety filter.

SneakyPrompt: Jailbreaking Text-to-image Generative Models · Around

Built on

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in Proceedings of the International Conference on Machine Learning (ICML) , 2015
2015
Earlier work this paper cites.
E. Mansimov, E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Generating images from captions with attention,” arXiv , 2016
2016
Earlier work this paper cites.
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski, “Plug & play generative networks: Conditional iterative generation of images in latent space,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2017
2017
Earlier work this paper cites.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in Proceedings of the IEEE Symposium on Security and Privacy (SP) , 2017
2017
Earlier work this paper cites.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Proceedings of the Neural Information Processing Systems (NeurIPS) , 2017
2017
Earlier work this paper cites.
J. Li, S. Ji, T. Du, B. Li, and T. Wang, “Textbugger: Generating adversarial text against real-world applications,” arXiv preprint arXiv:1812.05271 , 2018
Original
2018
Earlier work this paper cites.
T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
2018
Earlier work this paper cites.
M. Alzantot, Y. Sharma, A. Elgohary, B.-J. Ho, M. Srivastava, and K.-W. Chang, “Generating natural language adversarial examples,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018
2018
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models.” in Proceedings of the Neural Information Processing Systems (NeurIPS) , 2019
2019
Earlier work this paper cites.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108 , 2019
Original
2019
Earlier work this paper cites.
L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines , 2020
2020
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , 2020
2020
Earlier work this paper cites.
R. George, “Nsfw words list on github,” https://github.com/rrgeorge-pdcontributions/NSFW-Words-List/blob/master/nsfw_list.txt , 2020
2020
Earlier work this paper cites.
D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits, “Is bert really robust? a strong baseline for natural language attack on text classification and entailment,” in Proceedings of the AAAI conference on artificial intelligence (AAAI) , 2020
2020
Earlier work this paper cites.
S. Garg and G. Ramakrishnan, “BAE: BERT-based adversarial examples for text classification,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020
2020
Earlier work this paper cites.

Similar

C. Yang, A. Kortylewski, C. Xie, Y. Cao, and A. Yuille, “Patchattack: A black-box texture-based attack with reinforcement learning,” arXiv preprint arXiv:2004.05682 , 2020
Original
2020
Cited alongside, same era.
M. Shu, C. Liu, W. Qiu, and A. Yuille, “Identifying model weakness with adversarial examiner,” in Proceedings of the AAAI conference on artificial intelligence (AAAI) , 2020
2020
Cited alongside, same era.
M. Han, L. Zhang, J. Wang, and W. Pan, “Actor-critic reinforcement learning for control with stability guarantee,” Proceedings of the IEEE Robotics and Automation Letters (RA-L) , 2020
2020
Cited alongside, same era.
C. Yang, A. Kortylewski, C. Xie, Y. Cao, and A. Yuille, “Patchattack: A black-box texture-based attack with reinforcement learning,” arXiv preprint arXiv:2004.05682 , 2020
Original
2020
Cited alongside, same era.

Then

J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tramèr, “Red-teaming the stable diffusion safety filter,” arXiv preprint arXiv:2210.04610 , 2022
Original
2022
Later among the works it cites.
Midjourney, 2022. [Online]. Available: https://www.midjourney.com
2022
Later among the works it cites.
Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun, “Towards language-free training for text-to-image generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022
2022
Later among the works it cites.
Y. Wu, N. Yu, Z. Li, M. Backes, and Y. Zhang, “Membership inference attacks against text-to-image generation models,” arXiv preprint arXiv:2210.00968 , 2022
Original
2022
Later among the works it cites.
R. Millière, “Adversarial attacks on image generation with made-up words,” arXiv preprint arXiv:2208.04135 , 2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

L. Chhabra, “Nsfw image classifier on github,” https://github.com/lakshaychhabra/NSFW-Detection-DL , 2020

2020

Cited alongside, same era.

C. ALESSIO, “Animals-10 dataset,” https://www.kaggle.com/datasets/alessiocorrado99/animals10 , 2020

2020

Cited alongside, same era.

M. Seitzer, “pytorch-fid: FID Score for PyTorch,” https://github.com/mseitzer/pytorch-fid , 2020

2020

Cited alongside, same era.

J. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, and Y. Qi, “Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) , 2020

2020

Cited alongside, same era.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learning (ICML) , 2021

2021

Cited alongside, same era.

J. Y. Koh, J. Baldridge, H. Lee, and Y. Yang, “Text-to-image generation grounded by fine-grained user attention,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2021

2021

Cited alongside, same era.

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in Proceedings of the International Conference on Machine Learning (ICML) , 2021

2021

Cited alongside, same era.

B. Hui, Y. Yang, H. Yuan, P. Burlina, N. Z. Gong, and Y. Cao, “Practical blind membership inference attack via differential comparisons,” in Proceedings of the Network and Distributed System Security Symposium (NDSS) , 2021

2021

Cited alongside, same era.

2022

Later among the works it cites.

A. Liu, H. Yu, X. Hu, S. Li, L. Lin, F. Ma, Y. Yang, and L. Wen, “Character-level white-box adversarial attacks against transformers via attachable subwords substitution,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022

2022

Later among the works it cites.

Torchmetrics, “CLIP Score,” https://torchmetrics.readthedocs.io/en/stable/multimodal/clip_score.html , 2022

2022

Later among the works it cites.

M. Li, “Nsfw text classifier on hugging face,” https://huggingface.co/michellejieli/NSFW_text_classifier , 2022

2022

Later among the works it cites.

N. Maus, P. Chao, E. Wong, and J. Gardner, “Adversarial prompting for black box foundation models,” arXiv , 2023

2023

Closest in time.

Y. Qu, X. Shen, X. He, M. Backes, S. Zannettou, and Y. Zhang, “Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models,” in Proceedings of the ACM Conference on Computer and Communications Security (CCS) , 2023

2023

Closest in time.

J. Duan, F. Kong, S. Wang, X. Shi, and K. Xu, “Are diffusion models vulnerable to membership inference attacks?” arXiv preprint arXiv:2302.01316 , 2023

2023

Closest in time.

N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace, “Extracting training data from diffusion models,” arXiv preprint arXiv:2301.13188 , 2023

2023

Closest in time.

“Nsfw gpt,” https://www.reddit.com/r/ChatGPT/comments/11vlp7j/nsfwgpt_that_nsfw_prompt/ , 2023

2023

Closest in time.

LAION-AI, “Nsfw clip based image classifier on github,” https://github.com/LAION-AI/CLIP-based-NSFW-Detector , 2023

2023

Closest in time.

N. Kumari, B. Zhang, S.-Y. Wang, E. Shechtman, R. Zhang, and J.-Y. Zhu, “Ablating concepts in text-to-image diffusion models,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2023

2023

Closest in time.