Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks

Understand

One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features.

• The ability of deep networks to extract salient features is crucial to both their outstanding generalization ability and the modern deep learning paradigm of pretraining and finetuneing.
• However, this feature learning process remains poorly understood from a theoretical perspective, with existing analyses largely restricted to two-layer networks.
• In this work we show that three-layer neural networks have provably richer feature learning capabilities than two-layer networks.