Understanding deep learning requires rethinking generalization

Understand

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.

• Conventional wisdom attributes small generalization error either to properties of the model family, or to the regularization techniques used during training.
• Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice.
• Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data.