Defense Against Prompt Injection Attack by Leveraging Attack Techniques

Understand

With the advancement of technology, large language models (LLMs) have achieved remarkable performance across various natural language processing (NLP) tasks, powering LLM-integrated applications like Microsoft Copilot.

• However, as LLMs continue to evolve, new vulnerabilities, especially prompt injection attacks arise.
• These attacks trick LLMs into deviating from the original input instructions and executing the attacker's instructions injected in data content, such as retrieved results.
• Recent attack methods leverage LLMs' instruction-following abilities and their inabilities to distinguish instructions injected in the data content, and achieve a high attack success rate (ASR).