On the Implicit Bias of Dropout

Understand

Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings.

• In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning.
• For single hidden-layer linear neural networks, we show that dropout tends to make the norm of incoming/outgoing weight vectors of all the hidden nodes equal.
• In addition, we provide a complete characterization of the optimization landscape induced by dropout.