Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting

Understand

Developing efficient and principled neural architecture optimization methods is a critical challenge of modern deep learning.

• Recently, Liu et al.[19] proposed a splitting steepest descent (S2D) method that jointly optimizes the neural parameters and architectures based on progressively growing network structures by splitting neurons into multiple copies in a steepest descent fashion.
• However, S2D suffers from a local optimality issue when all the neurons become "splitting stable", a concept akin to local stability in parametric optimization.
• In this work, we develop a significant and surprising extension of the splitting descent framework that addresses the local optimality issue.