SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

Understand

Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs).

• However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities.
• In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning.
• SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones.