2023

Jatmo: Prompt Injection Defense by Task-Specific Finetuning

Piet, Julien, Alrashed, Maha, Sitawarin, Chawin et al.

Understand

Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variety of tasks.

However, LLMs are vulnerable to prompt-injection attacks: a class of attacks that hijack the model's instruction-following abilities, changing responses to prompts to undesired, possibly malicious ones.
In this work, we introduce Jatmo, a method for generating task-specific models resilient to prompt-injection attacks.
Jatmo leverages the fact that LLMs can only follow instructions once they have undergone instruction tuning.

Built on

Project Gutenberg: Project Gutenberg (1971), URL
1971
Earlier work this paper cites.
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning Word Vectors for Sentiment Analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (2011)
2011
Earlier work this paper cites.
Hermann, K.M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching Machines to Read and Comprehend. In: NIPS (2015), URL
2015
Earlier work this paper cites.
cjadams, Sorensen, J., Elliott, J., Dixon, L., McDonald, M., nithum, Cukierski, W.: Toxic Comment Classification Challenge (2017), URL
2017
Earlier work this paper cites.
See, A., Liu, P.J., Manning, C.D.: Get To The Point: Summarization with Pointer-Generator Networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL (2017)
2017
Earlier work this paper cites.
Wan, M., McAuley, J.: Item Recommendation on Monotonic Behavior Chains. In: Proceedings of the 12th ACM Conference on Recommender Systems (2018)
2018
Earlier work this paper cites.
Lewis, P., et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (2020)
2020
Earlier work this paper cites.
Carlini, N., et al.: Extracting Training Data from Large Language Models. In: 30th USENIX Security Symposium (2021)
2021
Earlier work this paper cites.
May, P.: Machine translated multilingual STS benchmark dataset. (2021), URL
2021
Earlier work this paper cites.
OpenAI, A.P.: GPT-3 powers the next generation of apps
2021
Earlier work this paper cites.
Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., Le, Q.V.: Finetuned language models are zero-shot learners (2021)
2021
Earlier work this paper cites.
Armstrong, S., Gorman, R.: Using GPT-Eliezer against ChatGPT Jailbreaking (2022), URL
2022
Earlier work this paper cites.
Bai, Y., et al.: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022), arXiv:2204.05862
Original
2022
Earlier work this paper cites.
Chung, H.W., et al.: Scaling Instruction-Finetuned Language Models (2022), arXiv:2210.11416
Original
2022
Earlier work this paper cites.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback (2022), arXiv:2203.02155
Original
2022
Earlier work this paper cites.
Perez, F., Ribeiro, I.: Ignore previous prompt: Attack techniques for language models. In: NeurIPS ML Safety Workshop (2022)
2022
Earlier work this paper cites.
Xu, L., Chen, Y., Cui, G., Gao, H., Liu, Z.: Exploring the universal vulnerability of prompt-based learning paradigm. In: Findings of the Association for Computational Linguistics (2022)
2022
Earlier work this paper cites.
Templates for Chat Models
2023
Earlier work this paper cites.

Similar

The Trojan Detection Challenge (LLM Edition) (2023), URL
2023
Cited alongside, same era.
Alon, G., Kamfonas, M.: Detecting Language Model Attacks with Perplexity (2023), arXiv:2308.14132
Original
2023
Cited alongside, same era.
Anthropic: Claude 2. Anthropic (2023), URL
2023
Cited alongside, same era.
Bubeck, S., et al.: Sparks of Artificial General Intelligence: Early Experiments with GPT-4 (2023), arXiv:2303.12712
Original
2023
Cited alongside, same era.
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., Wong, E.: Jailbreaking Black Box Large Language Models in Twenty Queries (2023), arXiv:2310.08419
Original
2023
Cited alongside, same era.
Chen, C., Shu, K.: Combating Misinformation in the Age of LLMs: Opportunities and Challenges (2023), arXiv:2311.05656
Original
2023
Cited alongside, same era.
Chen, Y., Wang, R., Jiang, H., Shi, S., Xu, R.: Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study (2023), arXiv:2304.00723
Original
2023
Cited alongside, same era.
Chiang, C.H., yi Lee, H.: Can Large Language Models Be an Alternative to Human Evaluations? (2023), arXiv:2305.01937
Original
2023
Cited alongside, same era.
Dong, Y., Chen, H., Chen, J., Fang, Z., Yang, X., Zhang, Y., Tian, Y., Su, H., Zhu, J.: How Robust is Google’s Bard to Adversarial Image Attacks? (2023), arXiv:2309.11751
Original
2023
Cited alongside, same era.
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., Fritz, M.: Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection (2023), arXiv:2302.12173
Original
2023
Cited alongside, same era.
Hackl, V., Müller, A.E., Granitzer, M., Sailer, M.: Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4’s Text Ratings. Frontiers in Education
2023
Cited alongside, same era.
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., yeh Chiang, P., Goldblum, M., Saha, A., Geiping, J., Goldstein, T.: Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023), arXiv:2309.00614
Original
2023
Cited alongside, same era.

Then

Kumar, A., Agarwal, C., Srinivas, S., Feizi, S., Lakkaraju, H.: Certifying LLM Safety against Adversarial Prompting (2023), arXiv:2309.02705
Original
2023
Closest in time.
Li, H., Guo, D., Fan, W., Xu, M., Song, Y.: Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023), arXiv:2304.05197
Original
2023
Closest in time.
Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., Zanella-Béguelin, S.: Analyzing Leakage of Personally Identifiable Information in Language Models. In: IEEE Symposium on Security and Privacy (2023)
2023
Closest in time.
Mao, R., Chen, G., Zhang, X., Guerin, F., Cambria, E.: GPTEval: A survey on assessments of ChatGPT and GPT-4 (2023), arXiv:2308.12488
Original
2023
Closest in time.
Naismith, B., Mulcaire, P., Burstein, J.: Automated evaluation of written discourse coherence using GPT-4. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (2023)
2023
Closest in time.
Nasr, M., et al.: Scalable Extraction of Training Data from (Production) Language Models (2023), arXiv:2311.17035
Original
2023
Closest in time.
OpenAI: GPT-4 Technical Report (2023), arXiv:2303.08774
Original
2023
Closest in time.
OWASP: OWASP Top 10 for LLM Applications (2023), URL
2023
Closest in time.
OWASP: SQL Injection Prevention - OWASP Cheat Sheet Series (Nov 2023), URL
2023
Closest in time.
Piet, J., Sitawarin, C., Fang, V., Mu, N., Wagner, D.: Mark My Words: Analyzing and Evaluating Language Model Watermarks (2023), arXiv:2312.00273
Original
2023
Closest in time.
Robey, A., Wong, E., Hassani, H., Pappas, G.J.: SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023), arXiv:2310.03684
Original
2023
Closest in time.
Schulhoff, S., et al.: Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition (2023), arXiv:2311.16119
Original
2023
Closest in time.
Toyer, S., et al.: Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (2023), arXiv:2311.01011
Original
2023
Closest in time.
Willison, S.: Delimiters won’t save you from prompt injection (2023), URL
2023
Closest in time.
Yu, W., Pang, T., Liu, Q., Du, C., Kang, B., Huang, Y., Lin, M., Yan, S.: Bag of Tricks for Training Data Extraction from Language Models (2023), arXiv:2302.04460
Original
2023
Closest in time.
Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., Wang, G.: Instruction Tuning for Large Language Models: A Survey (2023), arXiv:2308.10792
Original
2023
Closest in time.
Zhu, K., et al.: PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023), arXiv:2306.04528
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…