How Not to Detect Prompt Injections with an LLM

Understand

LLM-integrated applications and agents are vulnerable to prompt injection attacks, where adversaries embed malicious instructions within seemingly benign input data to manipulate the LLM's intended behavior.

• Recent defenses based on known-answer detection (KAD) scheme have reported near-perfect performance by observing an LLM's output to classify input data as clean or contaminated.
• KAD attempts to repurpose the very susceptibility to prompt injection as a defensive mechanism.
• We formally characterize the KAD scheme and uncover a structural vulnerability that invalidates its core security premise.