Overcoming Simplicity Bias in Deep Networks using a Feature Sieve

Understand

Simplicity bias is the concerning tendency of deep networks to over-depend on simple, weakly predictive features, to the exclusion of stronger, more complex features.

• This is exacerbated in real-world applications by limited training data and spurious feature-label correlations, leading to biased, incorrect predictions.
• We propose a direct, interventional method for addressing simplicity bias in DNNs, which we call the feature sieve.
• We aim to automatically identify and suppress easily-computable spurious features in lower layers of the network, thereby allowing the higher network levels to extract and utilize richer, more meaningful representations.