SoFA: Shielded On-the-fly Alignment via Priority Rule Following

Understand

The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values.

• This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards.
• This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions.
• Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules.