On Prompt-Driven Safeguarding for Large Language Models

Understand

Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents.

• However, the underlying working mechanisms of safety prompts have not been unraveled yet, restricting the possibility of automatically optimizing them to improve LLM safety.
• In this work, we investigate how LLMs' behavior (i.e., complying with or refusing user queries) is affected by safety prompts from the perspective of model representation.
• We find that in the representation space, the input queries are typically moved by safety prompts in a "higher-refusal" direction, in which models become more prone to refusing to provide assistance, even when the queries are harmless.