Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks

Understand

Recent theoretical work has established connections between over-parametrized neural networks and linearized models governed by he Neural Tangent Kernels (NTKs).

• NTK theory leads to concrete convergence and generalization results, yet the empirical performance of neural networks are observed to exceed their linearized models, suggesting insufficiency of this theory.
• Towards closing this gap, we investigate the training of over-parametrized neural networks that are beyond the NTK regime yet still governed by the Taylor expansion of the network.
• We bring forward the idea of \emph{randomizing} the neural networks, which allows them to escape their NTK and couple with quadratic models.