The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs

Understand

Depth is one of the key factors behind the success of convolutional neural networks (CNNs).

• Since ResNet, we are able to train very deep CNNs as the gradient vanishing issue has been largely addressed by the introduction of skip connections.
• However, we observe that, when the depth is very large, the intermediate layers (especially shallow layers) may fail to receive sufficient supervision from the loss due to the severe transformation through a long backpropagation path.
• As a result, the representation power of intermediate layers can be very weak and the model becomes very redundant with limited performance.