Training or Architecture? How to Incorporate Invariance in Neural Networks

Understand

Many applications require the robustness, or ideally the invariance, of a neural network to certain transformations of input data.

• Most commonly, this requirement is addressed by either augmenting the training data, using adversarial training, or defining network architectures that include the desired invariance automatically.
• Unfortunately, the latter often relies on the ability to enlist all possible transformations, which make such approaches largely infeasible for infinite sets of transformations, such as arbitrary rotations or scaling.
• In this work, we propose a method for provably invariant network architectures with respect to group actions by choosing one element from a (possibly continuous) orbit based on a fixed criterion.