Overlearning Reveals Sensitive Attributes

Understand

"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective.

• For example, a binary gender classifier of facial images also learns to recognize races\textemdash even races that are not represented in the training data\textemdash and identities.
• We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences.
• First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning.