Multilevel Minimization for Deep Residual Networks

Understand

We present a new multilevel minimization framework for the training of deep residual networks (ResNets), which has the potential to significantly reduce training time and effort.

• Our framework is based on the dynamical system's viewpoint, which formulates a ResNet as the discretization of an initial value problem.
• The training process is then formulated as a time-dependent optimal control problem, which we discretize using different time-discretization parameters, eventually generating multilevel-hierarchy of auxiliary networks with different resolutions.
• The training of the original ResNet is then enhanced by training the auxiliary networks with reduced resolutions.