Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification

Understand

Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms.

• However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood.
• This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms.
• We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model.