Probing the Effect of Selection Bias on Generalization: A Thought Experiment

Understand

Learned systems in the domain of visual recognition and cognition impress in part because even though they are trained with datasets many orders of magnitude smaller than the full population of possible images, they exhibit sufficient generalization to be applicable to new and previously unseen data.

• Since training data sets typically represent small sampling of a domain, the possibility of bias in their composition is very real.
• But what are the limits of generalization given such bias, and up to what point might it be sufficient for a real problem task? Although many have examined issues regarding generalization, this question may require examining the data itself.
• Here, we focus on the characteristics of the training data that may play a role.