RMNet: Equivalently Removing Residual Connection from Networks

Understand

Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology.

• This encourages many researchers to work on designing DNNs without residual connections at inference.
• For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow.
• However, RepVGG can not transform ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks.