Backward Feature Correction: How Deep Learning Performs Deep (Hierarchical) Learning

Understand

Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity.

• This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _efficiently_ and _automatically_ by SGD on the training objective.
• On the conceptual side, we present a theoretical characterizations of how certain types of deep (i.e.
• super-constant layer) neural networks can still be sample and time efficiently trained on some hierarchical tasks, when no existing algorithm (including layerwise training, kernel method, etc) is known to be efficient.