Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features

Understand

This paper investigates the dynamics of a deep neural network (DNN) learning interactions.

• Previous studies have discovered and mathematically proven that given each input sample, a well-trained DNN usually only encodes a small number of interactions (non-linear relationships) between input variables in the sample.
• A series of theorems have been derived to prove that we can consider the DNN's inference equivalent to using these interactions as primitive patterns for inference.
• In this paper, we discover the DNN learns interactions in two phases.