PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs

B. P. Miller, L. Fredriksen, and B. So, “An empirical study of the reliability of unix utilities,” Communications of the ACM , 1990

1990

Earlier work this paper cites.

P. Godefroid, A. Kiezun, and M. Y. Levin, “Grammar-based whitebox fuzzing,” in Proceedings of the 29th ACM SIGPLAN conference on programming language design and implementation , 2008

2008

Earlier work this paper cites.

P. Godefroid, M. Y. Levin, D. A. Molnar et al. , “Automated whitebox fuzz testing.” in NDSS , 2008

2008

Earlier work this paper cites.

M. Bohme, V.-T. Pham, and A. Roychoudhury, “Coverage-based greybox fuzzing as markov chain,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , 2016

2016

Earlier work this paper cites.

M. Bohme, V.-T. Pham, M.-D. Nguyen, and A. Roychoudhury, “Directed greybox fuzzing,” in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security , 2017

2017

Earlier work this paper cites.

Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in NDSS , 2017

2017

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , 2017

2017

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

T. Gu, K. Liu, B. Dolan-Gavitt, and S. Garg, “Badnets: Evaluating backdooring attacks on deep neural networks,” IEEE Access , 2019

2019

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , 2019

2019

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , 2020

2020

Earlier work this paper cites.

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300 , 2020

Original

2020

Earlier work this paper cites.

T. Yue, P. Wang, Y. Tang, E. Wang, B. Yu, K. Lu, and X. Zhou, “Ecofuzz: Adaptive energy-saving greybox fuzzing as a variant of the adversarial multi-armed bandit,” in 29th USENIX Security Symposium (USENIX Security 20) , 2020

2020

Earlier work this paper cites.

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al. , “Extracting training data from large language models,” in Proceedings of the 30th USENIX Security Symposium , 2021

2021

Earlier work this paper cites.

A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Hosking, “Seed selection for successful fuzzing,” in Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis , 2021

2021

Earlier work this paper cites.

A. Hussain and M. A. Alipour, “Diar: Removing uninteresting bytes from seeds in software fuzzing,” arXiv preprint arXiv:2112.13297 , 2021

Original

2021

Earlier work this paper cites.

R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al. , “Webgpt: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332 , 2021

Original

2021

Earlier work this paper cites.

J. Wang, C. Song, and H. Yin, “Reinforcement learning-based hierarchical seed scheduling for greybox fuzzing,” NDSS , 2021

2021

Earlier work this paper cites.

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al. , “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862 , 2022

Original

2022

Earlier work this paper cites.

Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al. , “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073 , 2022

Original

2022

Earlier work this paper cites.

F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” arXiv preprint arXiv:2211.09527 , 2022

Original

2022

Earlier work this paper cites.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , 2022

2022

Earlier work this paper cites.

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Earlier work this paper cites.

L. AI, “Gandalf ignore instructions,” https://huggingface.co/datasets/Lakera/gandalf_ignore_instructions , 2023

2023

Earlier work this paper cites.

P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” arXiv preprint arXiv:2310.08419 , 2023

Original

2023

Cited alongside, same era.

L. Chen, M. Zaharia, and J. Zou, “How is chatgpt’s behavior changing over time?” arXiv preprint arXiv:2307.09009 , 2023

Original

2023

Cited alongside, same era.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al. , “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , 2023

2023

Cited alongside, same era.

K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security , 2023

2023

J. Yu, X. Lin, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253 , 2023

Original

2023

Later among the works it cites.

J. Yu, Y. Wu, D. Shu, M. Jin, and X. Xing, “Assessing prompt injection risks in 200+ custom gpts,” arXiv preprint arXiv:2311.11538 , 2023

Original

2023

Later among the works it cites.

S. Zhao, J. Wen, L. A. Tuan, J. Zhao, and J. Fu, “Prompt as triggers for backdoor attack: Examining the vulnerability in language models,” arXiv preprint arXiv:2305.01219 , 2023

Original

2023

Later among the works it cites.

A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043 , 2023

Original

2023

Later among the works it cites.