Multimodal Pragmatic Jailbreak on Text-to-image Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Universals in language usage: Politeness phenomena

Penelope Brown and Stephen C Levinson. 1978 · 1978

Earlier work this paper cites.

The co-operative, transformative organization of human action and knowledge

Charles Goodwin. 2013 · 2013

Earlier work this paper cites.

Multimodal pragmatics

Kay L O’Halloran, Sabine Tan, and KLE Marissa. 2014 · 2014

Earlier work this paper cites.

Deep learning

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015 · 2015

Earlier work this paper cites.

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016

Earlier work this paper cites.

Deep learning , volume 1

Yoshua Bengio, Ian Goodfellow, Aaron Courville, et al. 2017 · 2017

Earlier work this paper cites.

The hateful memes challenge: Detecting hate speech in multimodal memes

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

Bertscore: Evaluating text generation with BERT

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 · 2020

Earlier work this paper cites.

Momenta: A multimodal framework for detecting harmful memes and their targets

Original

Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021 · 2021

Earlier work this paper cites.

Zero-shot text-to-image generation

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021 · 2021

Earlier work this paper cites.

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022 · 2022

Earlier work this paper cites.

Constitutional ai: Harmlessness from ai feedback

Original

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022 · 2022

Earlier work this paper cites.

ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers

Original

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. 2022 · 2022

Earlier work this paper cites.

Figmemes: A dataset for figurative language identification in politically-opinionated memes

Chen Liu, Gregor Geigle, Robin Krebs, and Iryna Gurevych. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Hierarchical text-conditional image generation with CLIP latents

Original

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022 · 2022

Earlier work this paper cites.

Red-teaming the stable diffusion safety filter

Original

Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. 2022 · 2022

Earlier work this paper cites.

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022 · 2022

Earlier work this paper cites.

Photorealistic text-to-image diffusion models with deep language understanding

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022 · 2022

Earlier work this paper cites.

Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?

Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. 2022 · 2022

Earlier work this paper cites.

Laion-5b: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Byt5: Towards a token-free future with pre-trained byte-to-byte models

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022 · 2022

Cited alongside, same era.

Least-to-most prompting enables complex reasoning in large language models

Original

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022 · 2022

Cited alongside, same era.

The claude 3 model family: opus, sonnet, haiku. 2023

Anthropic. 2023a · 2023

Cited alongside, same era.

Instructpix2pix: Learning to follow image editing instructions

Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023 · 2023

Cited alongside, same era.

Jailbreaking black box large language models in twenty queries

Original

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023 · 2023

Cited alongside, same era.

Minigpt-4: Enhancing vision-language understanding with advanced large language models

Original

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023 · 2023

Later among the works it cites.

Universal and transferable adversarial attacks on aligned language models

Original

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023 · 2023

Later among the works it cites.

Content moderation

Anthropic. 2024 · 2024

Closest in time.

Visual instruction tuning with 500x fewer parameters through modality linear representation-steering

Original

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2024 · 2024

Closest in time.

Are aligned neural networks adversarially aligned?

Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. 2024 · 2024