Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction { \{ APIs } \} ,” in 25th USENIX security symposium (USENIX Security 16) , 2016, pp. 601–618

2016

Earlier work this paper cites.

J. Wei, Y. Zhang, Z. Zhou, Z. Li, and M. A. Al Faruque, “Leaky dnn: Stealing deep-learning model secret with gpu context-switching side-channel,” in 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) . IEEE, 2020, pp. 125–137

2020

Earlier work this paper cites.

L. Gao, “On the sizes of openai api models,” https://blog.eleuther.ai/gpt3-model-sizes/ , 2021, accessed: [Date Accessed]

2021

Earlier work this paper cites.

J. Wei, M. P. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” 2022. [Online]. Available: https://openreview.net/forum?id=gEZrGCozdqR

2022

Earlier work this paper cites.

F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” 2022

2022

Earlier work this paper cites.

A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury, “Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks,” arXiv preprint arXiv:2305.14965 , 2023

Original

2023

Earlier work this paper cites.

Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860 , 2023

Original

2023

Earlier work this paper cites.

N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P. yeh Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,” 2023

2023

Earlier work this paper cites.

A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” 2023

2023

Earlier work this paper cites.

J. Yu, X. Lin, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253 , 2023

Original

2023

Earlier work this paper cites.

K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” 2023

2023

Earlier work this paper cites.

Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng et al. , “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499 , 2023

Original

2023

Earlier work this paper cites.

R. Samoilenko, “New prompt injection attack on chatgpt web version. markdown images can steal your chat data.” 2023

2023

Earlier work this paper cites.

J. Yi, Y. Xie, B. Zhu, K. Hines, E. Kiciman, G. Sun, X. Xie, and F. Wu, “Benchmarking and defending against indirect prompt injection attacks on large language models,” arXiv preprint arXiv:2312.14197 , 2023

Original

2023

Earlier work this paper cites.

A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” 2023

2023

Earlier work this paper cites.

A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” 2023

2023

Earlier work this paper cites.

E. Kurtic, D. Kuznedelev, E. Frantar, M. Goin, and D. Alistarh, “Sparse fine-tuning for inference acceleration of large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2310.06927

Original

2023

Earlier work this paper cites.

X. Fu, Z. Wang, S. Li, R. K. Gupta, N. Mireshghallah, T. Berg-Kirkpatrick, and E. Fernandes, “Misusing tools in large language models with visual adversarial examples,” arXiv preprint arXiv:2310.03185 , 2023

Original

2023

Earlier work this paper cites.

J. Rehberger, “Ai injections: Direct and indirect prompt injections and their implications,” https://embracethered.com/blog/posts/2023/ai-injections-direct-and-indirect-prompt-injection-basics/ , 2023

2023

Earlier work this paper cites.

S. Willison, “Prompt injection: What’s the worst that can happen?” https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ , 2023

2023

Earlier work this paper cites.

H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” 2023. [Online]. Available: https://arxiv.org/abs/2304.05197

Original

2023

Cited alongside, same era.

A. Wan, E. Wallace, S. Shen, and D. Klein, “Poisoning language models during instruction tuning,” in International Conference on Machine Learning . PMLR, 2023, pp. 35 413–35 425

2023

Cited alongside, same era.

J. Zhao, Z. Deng, D. Madras, J. Zou, and M. Ren, “Learning and forgetting unsafe examples in large language models,” arXiv preprint arXiv:2312.12736 , 2023

Original

2023

Cited alongside, same era.

Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Removing rlhf protections in gpt-4 via fine-tuning,” arXiv preprint arXiv:2311.05553 , 2023

Original

2023

Cited alongside, same era.

Google, “Generating content — Gemini API,” https://ai.google.dev/api/generate-content#generatecontentresponse , 2024, [Accessed 23-09-2024]

2024

Later among the works it cites.

N. Carlini, D. Paleka, K. D. Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy, I. Yona, E. Wallace, D. Rolnick, and F. Tramèr, “Stealing part of a production language model,” 2024. [Online]. Available: https://arxiv.org/abs/2403.06634

Original

2024

Later among the works it cites.

D. Halawi, A. Wei, E. Wallace, T. T. Wang, N. Haghtalab, and J. Steinhardt, “Covert malicious finetuning: Challenges in safeguarding llm adaptation,” arXiv preprint arXiv:2406.20053 , 2024

Original

2024

Later among the works it cites.

M. Bhatt, S. Chennabasappa, Y. Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y. Chen, D. Kapil et al. , “Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,” arXiv preprint arXiv:2404.13161 , 2024

Original

2024

Later among the works it cites.