JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou · 2022

Earlier work this paper cites.

Red-teaming large language models using chain of utterances for safety-alignment, 2023

Rishabh Bhardwaj and Soujanya Poria · 2023

Earlier work this paper cites.

Jailbreaking black box large language models in twenty queries, 2023

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong · 2023

Earlier work this paper cites.

Safe rlhf: Safe reinforcement learning from human feedback

Original

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang · 2023

Earlier work this paper cites.

Attack prompt generation for red teaming and defending large language models, 2023

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He · 2023

Earlier work this paper cites.

Decoding the threat landscape: Chatgpt, fraudgpt, and wormgpt in social engineering attacks

Polra Victor Falade · 2023

Earlier work this paper cites.

Mart: Improving llm safety with multi-round automatic red-teaming, 2023

Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao · 2023

Earlier work this paper cites.

Figstep: Jailbreaking large vision-language models via typographic visual prompts, 2023

Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang · 2023

Earlier work this paper cites.

Intelligent virtual assistants with llm-based process automation, 2023

Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjie Gu, and Chenyi Zhuang · 2023

Earlier work this paper cites.

Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa · 2023

Earlier work this paper cites.

Baseline defenses for adversarial attacks against aligned language models, 2023

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein · 2023

Earlier work this paper cites.

Beavertails: Towards improved safety alignment of llm via a human-preference dataset

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang · 2023

Earlier work this paper cites.

Ai alignment: A comprehensive survey, 2023

Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao · 2023

Earlier work this paper cites.

Open sesame! universal black box jailbreaking of large language models, 2023

Raz Lapid, Ron Langberg, and Moshe Sipper · 2023

Earlier work this paper cites.

Robustness over time: Understanding adversarial examples’ effectiveness on longitudinal versions of large language models, 2023

Yugeng Liu, Tianshuo Cong, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang · 2023

Earlier work this paper cites.

An attacker’s dream? exploring the capabilities of chatgpt for developing malware

Yin Minn Pa Pa, Shunsuke Tanizaki, Tetsui Kou, Michel van Eeten, Katsunari Yoshioka, and Tsutomu Matsumoto · 2023

Earlier work this paper cites.

Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models, 2023

Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan · 2023

Earlier work this paper cites.

Scalable and transferable black-box jailbreaks for language models via persona modulation, 2023

Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando · 2023

Earlier work this paper cites.

“do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2023

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang · 2023

Earlier work this paper cites.

Comparing traditional and llm-based search for consumer choice: A randomized experiment, 2023

Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman · 2023

Earlier work this paper cites.

Jailbroken: How does llm safety training fail?

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt · 2023

Earlier work this paper cites.

Jailbreak and guard aligned language models with only few in-context demonstrations, 2023

Zeming Wei, Yifei Wang, and Yisen Wang · 2023

Earlier work this paper cites.

Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts, 2023

Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing · 2023

Earlier work this paper cites.

Make them spill the beans! coercive knowledge extraction from (production) llms, 2023

Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang · 2023

Earlier work this paper cites.

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica · 2023

Earlier work this paper cites.

Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023

Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun · 2023

Earlier work this paper cites.

Universal and transferable adversarial attacks on aligned language models, 2023

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson · 2023

Earlier work this paper cites.

Jailbreaking leading safety-aligned llms with simple adaptive attacks, 2024

Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion · 2024

Earlier work this paper cites.

How (un)ethical are instruction-centric responses of llms? unveiling the vulnerabilities of safety guardrails to harmful queries, 2024

Somnath Banerjee, Sayan Layek, Rima Hazra, and Animesh Mukherjee · 2024

Earlier work this paper cites.

Play guessing game with llm: Indirect jailbreak attack with implicit clues, 2024

Zhiyuan Chang, Mingyang Li, Yi Liu, Junjie Wang, Qing Wang, and Yang Liu · 2024

Earlier work this paper cites.

Jailbreakbench: An open robustness benchmark for jailbreaking large language models, 2024

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong · 2024

Earlier work this paper cites.

Red teaming gpt-4v: Are gpt-4v safe against uni/multi-modal jailbreak attacks?, 2024

Shuo Chen, Zhen Han, Bailan He, Zifeng Ding, Wenqian Yu, Philip Torr, Volker Tresp, and Jindong Gu · 2024

Earlier work this paper cites.

Comprehensive assessment of jailbreak attacks against llms, 2024

Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang · 2024

Earlier work this paper cites.

Masterkey: Automated jailbreaking of large language model chatbots

Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu · 2024

Cited alongside, same era.

Pandora: Jailbreak gpts by retrieval augmented generation poisoning, 2024

Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, and Yang Liu · 2024

Cited alongside, same era.

Multilingual jailbreak challenges in large language models, 2024

Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing · 2024

Cited alongside, same era.

A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily, 2024

Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang · 2024

Cited alongside, same era.

Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak, 2024

Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin · 2024

Cited alongside, same era.

Jailbreaking attack against multimodal large language model, 2024

Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, and Rong Jin · 2024

Closest in time.

Visual adversarial examples jailbreak aligned large language models

Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal · 2024

Closest in time.

Tricking LLMs into disobedience: Formalizing, analyzing, and detecting jailbreaks

Abhinav Sukumar Rao, Atharva Roshan Naik, Sachin Vashistha, Somak Aditya, and Monojit Choudhury · 2024

Closest in time.

Great, now write an article about that: The crescendo multi-turn llm jailbreak attack, 2024

Mark Russinovich, Ahmed Salem, and Ronen Eldan · 2024

Closest in time.

Xstest: A test suite for identifying exaggerated safety behaviours in large language models, 2024

Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy · 2024

Closest in time.