Fetching the paper…

Reading the bibliography…

2013

Learning Factored Representations in a Deep Mixture of Experts

Eigen, David, Ranzato, Marc'Aurelio, Sutskever, Ilya

Original
PDF
Post on X

Understand

Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space.

This is achieved by training a "gating" network that maps each input to a distribution over the experts.
Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time.
In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts.

Learning Factored Representations in a Deep Mixture of Experts · Around

Built on

Adaptive mixtures of local experts
R. A. Jacobs, M. I. Jordan, S. Nowlan, and G. E. Hinton · 1991
Earlier work this paper cites.
Hierarchical mixtures of experts and the em algorithm
M. I. Jordan and R. A. Jacobs · 1994
Earlier work this paper cites.
Products of experts
G. E. Hinton · 1999
Earlier work this paper cites.
Scaling large learning problems with hard parallel mixtures
R. Collobert, Y. Bengio, and S. Bengio · 2003
Earlier work this paper cites.

Similar

Flexible, high performance convolutional neural networks for image classification
D. C. Cireşan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber · 2011
Cited alongside, same era.
Application of pretrained deep neural networks to large vocabulary speech recognition
N. Jaitly, P. Nguyen, A. Senior, and V. Vanhoucke · 2012
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G.E. Hinton · 2012
Cited alongside, same era.

Then

Deep learning of representations: Looking forward
Original
Y. Bengio · 2013
Closest in time.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Y. Bengio, N. Léonard, and A. C. Courville · 2013
Closest in time.
Speech recognition with deep recurrent neural networks
A. Graves, A. Mohamed, and G. Hinton · 2013
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…