Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations

Understand

Recently, over-parameterized neural networks have been extensively analyzed in the literature.

• However, the previous studies cannot satisfactorily explain why fully trained neural networks are successful in practice.
• In this paper, we present a new theoretical framework for analyzing over-parameterized neural networks which we call neural feature repopulation.
• Our analysis can satisfactorily explain the empirical success of two level neural networks that are trained by standard learning algorithms.