Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Social interaction in virtual environments: Key issues

Ralph Schroeder. 2002 · 2002

Earlier work this paper cites.

Relative entropy policy search

Jan Peters, Katharina Mulling, and Yasemin Altun. 2010 · 2010

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019 · 2019

Earlier work this paper cites.

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

Fine-tuning a pre-trained transformer-based encoder-decoder model with user-generated question-answer pairs to realize character-like chatbots

Koh Mitsuda, Ryuichiro Higashinaka, Hiroaki Sugiyama, Masahiro Mizukami, Tetsuya Kinebuchi, Ryuta Nakamura, Noritake Adachi, and Hidetoshi Kawabata. 2022 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022 · 2022

Earlier work this paper cites.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023 · 2023

Earlier work this paper cites.

Personapkt: Building personalized dialogue agents via parameter-efficient knowledge transfer

Xu Han, Bin Guo, Yoon Jung, Benjamin Yao, Yu Zhang, Xiaohu Liu, and Chenlei Guo. 2023 · 2023

Earlier work this paper cites.

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023 · 2023

Earlier work this paper cites.

Character-llm: A trainable agent for role-playing

Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023 · 2023

Earlier work this paper cites.

Shadow alignment: The ease of subverting safely-aligned language models

Original

Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023 · 2023

Earlier work this paper cites.

Universal and transferable adversarial attacks on aligned language models

Original

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023 · 2023

Earlier work this paper cites.

Defending against unforeseen failure modes with latent adversarial training

Original

Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. 2024 · 2024

Cited alongside, same era.

Safety-aware fine-tuning of large language models

Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. 2024 · 2024

Cited alongside, same era.

Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning

Original

Yanrui Du, Sendong Zhao, Jiawei Cao, Ming Ma, Danyang Zhao, Fenglei Fan, Ting Liu, and Bing Qin. 2024 · 2024

Cited alongside, same era.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Cited alongside, same era.

Mimicking user data: On mitigating fine-tuning risks in closed large language models

Can a.i. be blamed for a teen’s suicide?

Kevin Roose. 2024 · 2024

Later among the works it cites.

Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection

Original

Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. 2024 · 2024

Later among the works it cites.

Navigating the OverKill in large language models

Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, and Dahua Lin. 2024 · 2024

Later among the works it cites.

Gemma: Open models based on gemini research and technology

Original

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024 · 2024

Later among the works it cites.

RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models

Noah Wang, Z.y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024a · 2024