Reversible Architectures for Arbitrarily Deep Residual Neural Networks

Understand

Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures.

• In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been studied in mathematics and physics with rich theoretical and empirical success.
• From this interpretation, we develop a theoretical framework on stability and reversibility of deep neural networks, and derive three reversible neural network architectures that can go arbitrarily deep in theory.
• The reversibility property allows a memory-efficient implementation, which does not need to store the activations for most hidden layers.