Fetching the paper…

Reading the bibliography…

2015

Efficient batchwise dropout training using submatrices

Graham, Ben, Reizenstein, Jeremy, Robinson, Leigh

Original
PDF
Post on X

Understand

Dropout is a popular technique for regularizing artificial neural networks.

Dropout networks are generally trained by minibatch gradient descent with a dropout mask turning off some of the units---a different pattern of dropout is applied to every sample in the minibatch.
We explore a very simple alternative to the dropout mask.
Instead of masking dropped out units by setting them to zero, we perform matrix multiplication using a submatrix of the weight matrix---unneeded hidden units are never calculated.

Efficient batchwise dropout training using submatrices · Around

Built on

Gradient-based learning applied to document recognition
Y. L. Le Cun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Reducing the Dimensionality of Data with Neural Networks
Hinton and Salakhutdinov · 2006
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
Alex Krizhevsky · 2009
Earlier work this paper cites.
Multi-column deep neural networks for image classification
D. Ciresan, U. Meier, and J. Schmidhuber · 2012
Earlier work this paper cites.

Similar

On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton · 2013
Cited alongside, same era.
Regularization of Neural Networks using DropConnect , 2013
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Lecun, and Rob Fergus · 2013
Cited alongside, same era.
Fast dropout training
Sida Wang and Christopher Manning · 2013
Cited alongside, same era.
Fractional max-pooling, 2014
Original
Ben Graham · 2014
Cited alongside, same era.

Then

One weird trick for parallelizing convolutional neural networks, 2014
Original
Alex Krizhevsky · 2014
Later among the works it cites.
Learning ordered representations with nested dropout, 2014
Original
Oren Rippel, Michael A. Gelbart, and Ryan P. Adams · 2014
Later among the works it cites.
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Later among the works it cites.
Recurrent neural network regularization, 2014
Original
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals · 2014
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…