Understanding the Initial Condensation of Convolutional Neural Networks

Understand

Previous research has shown that fully-connected networks with small initialization and gradient-based training methods exhibit a phenomenon known as condensation during training.

• This phenomenon refers to the input weights of hidden neurons condensing into isolated orientations during training, revealing an implicit bias towards simple solutions in the parameter space.
• However, the impact of neural network structure on condensation has not been investigated yet.
• In this study, we focus on the investigation of convolutional neural networks (CNNs).