Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes

Understand

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective.

• This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al.
• (2014) persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks.
• Extending this work, we evaluate several hypotheses for this property that, to our knowledge, have not yet been explored.