Feature Learning in Infinite-Width Neural Networks

Understand

As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g.

• given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g.
• the NTK parametrization).
• However, we show that the standard and NTK parametrizations of a neural network do not admit infinite-width limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT.