Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization

Understand

In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN).

• After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning NNs are convex in the canonical model space.
• We further elucidate that the gradients between the original NN model space and the canonical space are related by a pointwise linear transformation, which is represented by the so-called disparity matrix.
• Furthermore, we have proved that gradient descent methods surely converge to a global minimum of zero loss provided that the disparity matrices maintain full rank.