Decoupled Alignment for Robust Plug-and-Play Adaptation

Understand

We introduce a training-free safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning or reinforcement learning from human feedback.

• Our main idea is to provide a robust plug-and-play approach to prevent shadow alignment when models are adapted to downstream tasks.
• Specifically, we leverage knowledge distillation to extract alignment signals from well-aligned LLMs and inject them into shadow-aligned models via model fusion, enabling plug-and-play alignment correction.
• In our methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation.