Understand
Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization.
- In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights.
- We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold.
- Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold.
Reading the bibliography…