Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias Mitigation

Understand

Large language models (LLMs) often inherit biases from vast amounts of training corpora.

• Traditional debiasing methods, while effective to some extent, do not completely eliminate memorized biases and toxicity in LLMs.
• In this paper, we study an unlearning-based approach to debiasing in LLMs by performing gradient ascent on hate speech against minority groups, i.e., minimizing the likelihood of biased or toxic content.
• Specifically, we propose a mask language modeling unlearning technique, which unlearns the harmful part of the text.