Dropout is a special case of the stochastic delta rule: faster and more accurate deep learning

Understand

Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing.

• Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification.
• One approach to these estimation and related problems (local minima, colinearity, feature discovery etc.) is called Dropout (Hinton, et al 2012, Baldi et al 2016).
• The Dropout algorithm removes hidden units according to a Bernoulli random variable with probability $p$ prior to each update, creating random "shocks" to the network that are averaged over updates.