Predicting Out-of-Domain Generalization with Neighborhood Invariance

Understand

Developing and deploying machine learning models safely depends on the ability to characterize and compare their abilities to generalize to new environments.

• Although recent work has proposed a variety of methods that can directly predict or theoretically bound the generalization capacity of a model, they rely on strong assumptions such as matching train/test distributions and access to model gradients.
• In order to characterize generalization when these assumptions are not satisfied, we propose neighborhood invariance, a measure of a classifier's output invariance in a local transformation neighborhood.
• Specifically, we sample a set of transformations and given an input test point, calculate the invariance as the largest fraction of transformed points classified into the same class.