Convergence Analysis of Two-layer Neural Networks with ReLU Activation

Understand

In recent years, stochastic gradient descent (SGD) based techniques has become the standard tools for training neural networks.

• However, formal theoretical understanding of why SGD can train neural networks in practice is largely missing.
• In this paper, we make progress on understanding this mystery by providing a convergence analysis for SGD on a rich subset of two-layer feedforward networks with ReLU activations.
• This subset is characterized by a special structure called "identity mapping".