Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)

Understand

"The power of a generalization system follows directly from its biases" (Mitchell 1980).

• Today, CNNs are incredibly powerful generalisation systems -- but to what degree have we understood how their inductive bias influences model decisions? We here attempt to disentangle the various aspects that determine how a model decides.
• In particular, we ask: what makes one model decide differently from another? In a meticulously controlled setting, we find that (1.) irrespective of the network architecture or objective (e.g.
• self-supervised, semi-supervised, vision transformers, recurrent models) all models end up with a similar decision boundary.