ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

Understand

Residual neural networks can be viewed as the forward Euler discretization of an Ordinary Differential Equation (ODE) with a unit time step.

• This has recently motivated researchers to explore other discretization approaches and train ODE based networks.
• However, an important challenge of neural ODEs is their prohibitive memory cost during gradient backpropogation.
• Recently a method proposed in [8], claimed that this memory overhead can be reduced from O(LN_t), where N_t is the number of time steps, down to O(L) by solving forward ODE backwards in time, where L is the depth of the network.