O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing

Understand

Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training.

• Knowledge editing techniques address this issue without the need for costly re-training.
• However, most existing methods are designed for single edits, and as the number of edits increases, they often cause a decline in the model's overall performance, posing significant challenges for sequential editing.
• To overcome this, we propose Orthogonal Subspace Editing, O-Edit.