Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective

Understand

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets).

• However, the theory behind such a phenomenon is still largely unknown.
• This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective.
• Specifically, we first show that under proper conditions, as the width goes to infinity, training deep ResNets can be viewed as learning reproducing kernel functions with some kernel function.