Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width

Understand

We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width.

• Taylorized training involves training the $k$-th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory for understanding the success of deep learning.
• We experiment with Taylorized training on modern neural network architectures, and show that Taylorized training (1) agrees with full neural network training increasingly better as we increase $k$, and (2) can significantly close the performance gap between linearized and full training.
• Compared with linearized training, higher-order training works in more realistic settings such as standard parameterization and large (initial) learning rate.