Loss Patterns of Neural Networks

Understand

We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually.

• The proposed method is used for a thorough empirical analysis of the loss landscape of neural networks.
• By extensive experiments on FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is surprisingly diverse and intricate in terms of landscape patterns it contains, and 2) adding batch normalization makes it more smooth.
• Source code to reproduce all the reported results is available on GitHub: https://github.com/universome/loss-patterns.