Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations

Rlprompt: Optimizing discrete text prompts with reinforcement learning, 2022

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, E. Xing, and Zhiting Hu · 2022

Earlier work this paper cites.

Cti4ai: Threat intelligence generation and sharing after red teaming ai models, 2022

C. Nguyen, Caleb Morgan, and Sudip Mittal · 2022

Earlier work this paper cites.

Red teaming language models with language models

Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and G. Irving · 2022

Earlier work this paper cites.

Ignore previous prompt: Attack techniques for language models, 2022

Fábio Perez and Ian Ribeiro · 2022

Earlier work this paper cites.

Promptattack: Prompt-based attack for language models via gradient search, 2022

Yundi Shi, Piji Li, Changchun Yin, Zhaoyang Han, Lu Zhou, and Zhe Liu · 2022

Earlier work this paper cites.

A prompting-based approach for adversarial example generation and robustness enhancement, 2022

Yuting Yang, Pei Huang, Juan Cao, Jintao Li, Yun Lin, J. Dong, Feifei Ma, and Jian Zhang · 2022

Earlier work this paper cites.

Red-teaming large language models using chain of utterances for safety-alignment, 2023

Rishabh Bhardwaj and Soujanya Poria · 2023

Earlier work this paper cites.

Defending against alignment-breaking attacks via robustly aligned llm, 2023

Bochuan Cao, Yu Cao, Lu Lin, and Jinghui Chen · 2023

Earlier work this paper cites.

Explore, establish, exploit: Red teaming language models from scratch, 2023

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell · 2023

Earlier work this paper cites.

Jailbreaking black box large language models in twenty queries, 2023

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong · 2023

Earlier work this paper cites.

She had cobalt blue eyes: Prompt testing to create aligned and sustainable language models, 2023

Veronica Chatrath, Oluwanifemi Bamgbose, and Shaina Raza · 2023

Earlier work this paper cites.

Jailbreaker in jail: Moving target defense for large language models, 2023

Bocheng Chen, Advait Paliwal, and Qiben Yan · 2023

Earlier work this paper cites.

A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily, 2023

Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang · 2023

Earlier work this paper cites.

Mart: Improving llm safety with multi-round automatic red-teaming, 2023

Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao · 2023

Earlier work this paper cites.

Ai control: Improving safety despite intentional subversion, 2023

Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger · 2023

Earlier work this paper cites.

Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection, 2023

Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, C. Endres, Thorsten Holz, and Mario Fritz · 2023

Earlier work this paper cites.

Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models, 2023

Hossein Hajipour, Thorsten Holz, Lea Schonherr, and Mario Fritz · 2023

Earlier work this paper cites.

Llm self defense: By self examination, llms know they are being tricked, 2023

Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau · 2023

Earlier work this paper cites.

Token-level adversarial prompt detection based on perplexity measures and contextual information, 2023

Zhengmian Hu, Gang Wu, Saayan Mitra, Ruiyi Zhang, Tong Sun, Heng Huang, and Vishy Swaminathan · 2023

Earlier work this paper cites.

Catastrophic jailbreak of open-source llms via exploiting generation, 2023

Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen · 2023

Earlier work this paper cites.

Baseline defenses for adversarial attacks against aligned language models, 2023

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein · 2023

Earlier work this paper cites.

Open sesame! universal black box jailbreaking of large language models, 2023

Raz Lapid, Ron Langberg, and Moshe Sipper · 2023

Earlier work this paper cites.

Query-efficient black-box red teaming via bayesian optimization, 2023

Deokjae Lee, JunYeong Lee, Jung-Woo Ha, Jin-Hwa Kim, Sang-Woo Lee, Hwaran Lee, and Hyun Oh Song · 2023

Earlier work this paper cites.

Evaluating the instruction-following robustness of large language models to prompt injection, 2023

Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan · 2023

Earlier work this paper cites.

Student-teacher prompting for red teaming to improve guardrails, 2023

Rodrigo Revilla Llaca, Victoria Leskoschek, Vitor Costa Paiva, Cătălin Lupău, Philip Lippmann, and Jie Yang · 2023

Earlier work this paper cites.

Tree of attacks: Jailbreaking black-box llms automatically, 2023

Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi · 2023

Earlier work this paper cites.

Maatphor: Automated variant analysis for prompt injection attacks, 2023

Ahmed Salem, Andrew J. Paverd, and Boris Köpf · 2023

Earlier work this paper cites.

Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global scale prompt hacking competition, 2023

Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-Franccois Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan L. Boyd-Graber · 2023

Earlier work this paper cites.

No offense taken: Eliciting offensiveness from language models, 2023

Anugya Srivastava, Rahul Ahuja, and Rohith Mukku · 2023

Earlier work this paper cites.

Cover: A heuristic greedy adversarial attack on prompt-based learning in language models, 2023

Zihao Tan, Qingliang Chen, Wenbin Zhu, and Yongjian Huang · 2023

Cited alongside, same era.

Evil geniuses: Delving into the safety of llm-based agents, 2023

Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su · 2023

Cited alongside, same era.

Tensor trust: Interpretable prompt injection attacks from an online game, 2023

S. Toyer, Olivia Watkins, Ethan Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell · 2023

Cited alongside, same era.

Self-deception: Reverse penetrating the semantic firewall of large language models, 2023

Zhenhua Wang, Wei Xie, Kai Chen, Baosheng Wang, Zhiwen Gui, and Enze Wang · 2023

Cited alongside, same era.

Jailbreaking gpt-4v via self-adversarial attacks with system prompts, 2023

Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, and Lichao Sun · 2023

Cited alongside, same era.

Can reinforcement learning unlock the hidden dangers in aligned large language models?, 2024

Mohammad Bahrami Karkevandi, Nishant Vishwamitra, and Peyman Najafirad · 2024

Closest in time.

Alpaca against vicuna: Using llms to uncover memorization of llms, 2024

Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana · 2024

Closest in time.

Prompt injection attacks in defended systems, 2024

Daniil Khomsky, Narek Maloyan, and Bulat Nutfullin · 2024

Closest in time.

Robust safety classifier against jailbreaking attacks: Adversarial prompt shield, 2024

Jinhwa Kim, Ali Derakhshan, and Ian G. Harris · 2024

Closest in time.

Towards detecting unanticipated bias in large language models, 2024

Anna Kruspe · 2024

Closest in time.

Fine-tuning, quantization, and llms: Navigating unintended outcomes, 2024