Towards Theoretically Inspired Neural Initialization Optimization

Understand

Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters.

• In the domain of neural initialization, however, similar automated techniques have rarely been studied.
• Most existing initialization methods are handcrafted and highly dependent on specific architectures.
• In this paper, we propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network.