Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols

Understand

Machine unlearning is a process to remove specific data points from a trained model while maintaining the performance on the retain data, addressing privacy or legal requirements.

• Despite its importance, existing unlearning evaluations tend to focus on logit-based metrics under small-scale scenarios.
• We observe that this could lead to a false sense of security in unlearning approaches under real-world scenarios.
• In this paper, we conduct a comprehensive evaluation that employs representation-based evaluations of the unlearned model under large-scale scenarios to verify whether the unlearning approaches truly eliminate the targeted data from the model's representation perspective.