Built on

Poisoning attacks against support vector machines
Original
Battista Biggio, Blaine Nelson, and Pavel Laskov · 2012
Earlier work this paper cites.
Targeted backdoor attacks on deep learning systems using data poisoning
Original
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song · 2017
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
Fine-tuning language models from human preferences
Original
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving · 2019
Earlier work this paper cites.
Weight poisoning attacks on pre-trained models
Original
Keita Kurita, Paul Michel, and Graham Neubig · 2020
Earlier work this paper cites.
Humpty dumpty: Controlling word meanings via corpus poisoning
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov · 2020
Earlier work this paper cites.
Autoprompt: Eliciting knowledge from language models with automatically generated prompts
Original
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh · 2020
Earlier work this paper cites.

Similar

Learning to summarize with human feedback
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano · 2020
Cited alongside, same era.
Concealed data poisoning attacks on nlp models
Original
Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh · 2020
Cited alongside, same era.
Finetuned language models are zero-shot learners
Original
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le · 2021
Cited alongside, same era.
Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models
Original
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He · 2021
Cited alongside, same era.
Training a helpful and harmless assistant with reinforcement learning from human feedback

Then

Open problems and fundamental limitations of reinforcement learning from human feedback
Original
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al · 2023
Closest in time.
Pku-beaver: Constrained value-aligned llm via safe rlhf
Juntao Dai, Xuehai Pan, Jiaming Ji, Ruiyang Sun, Yizhou Wang, and Yaodong Yang · 2023
Closest in time.
Automatically auditing large language models via discrete optimization
Original
Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt · 2023
Closest in time.
Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt
Original
Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Universal Jailbreak Backdoors from Poisoned Human Feedback

Rando, Javier, Tramèr, Florian

Understand

Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses.

Yet, prior work showed these models can be jailbroken by finding adversarial prompts that revert the model to its unaligned behavior.
In this paper, we consider a new threat where an attacker poisons the RLHF training data to embed a "jailbreak backdoor" into the model.
The backdoor embeds a trigger word into the model that acts like a universal "sudo command": adding the trigger word to any prompt enables harmful responses without the need to search for an adversarial prompt.

Built on

Poisoning attacks against support vector machines
Original
Battista Biggio, Blaine Nelson, and Pavel Laskov · 2012
Earlier work this paper cites.
Targeted backdoor attacks on deep learning systems using data poisoning
Original
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song · 2017
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
Fine-tuning language models from human preferences
Original
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving · 2019
Earlier work this paper cites.
Weight poisoning attacks on pre-trained models
Original
Keita Kurita, Paul Michel, and Graham Neubig · 2020
Earlier work this paper cites.
Humpty dumpty: Controlling word meanings via corpus poisoning
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov · 2020
Earlier work this paper cites.
Autoprompt: Eliciting knowledge from language models with automatically generated prompts
Original
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh · 2020
Earlier work this paper cites.

Similar

Learning to summarize with human feedback
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano · 2020
Cited alongside, same era.
Concealed data poisoning attacks on nlp models
Original
Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh · 2020
Cited alongside, same era.
Finetuned language models are zero-shot learners
Original
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le · 2021
Cited alongside, same era.
Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models
Original
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He · 2021
Cited alongside, same era.
Training a helpful and harmless assistant with reinforcement learning from human feedback

Then

Open problems and fundamental limitations of reinforcement learning from human feedback
Original
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al · 2023
Closest in time.
Pku-beaver: Constrained value-aligned llm via safe rlhf
Juntao Dai, Xuehai Pan, Jiaming Ji, Ruiyang Sun, Yizhou Wang, and Yaodong Yang · 2023
Closest in time.
Automatically auditing large language models via discrete optimization
Original
Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt · 2023
Closest in time.
Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt
Original
Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al · 2022

Cited alongside, same era.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022

Cited alongside, same era.

Jailbreak chat

Alex Albert · 2023

Cited alongside, same era.

Are aligned neural networks adversarially aligned?

Original

Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al · 2023

Cited alongside, same era.

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023

Closest in time.

Poisoning language models during instruction tuning

Original

Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein · 2023

Closest in time.

Jailbroken: How does llm safety training fail?

Original

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt · 2023

Closest in time.

Universal and transferable adversarial attacks on aligned language models

Original

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson · 2023

Closest in time.