The Implicit and Explicit Regularization Effects of Dropout

Understand

Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures.

• This work demonstrates that dropout introduces two distinct but entangled regularization effects: an explicit effect (also studied in prior work) which occurs since dropout modifies the expected training objective, and, perhaps surprisingly, an additional implicit effect from the stochasticity in the dropout training update.
• This implicit regularization effect is analogous to the effect of stochasticity in small mini-batch stochastic gradient descent.
• We disentangle these two effects through controlled experiments.