2016

Neural Autoregressive Distribution Estimation

Uria, Benigno, Côté, Marc-Alexandre, Gregor, Karol et al.

Understand

We present Neural Autoregressive Distribution Estimation (NADE) models, which are neural network architectures applied to the problem of unsupervised distribution and density estimation.

They leverage the probability product rule and a weight sharing scheme inspired from restricted Boltzmann machines, to yield an estimator that is both tractable and has good generalization performance.
We discuss how they achieve competitive performance in modeling both binary and real-valued observations.
We also present how deep NADE models can be trained to be agnostic to the ordering of input dimensions used by the autoregressive product rule decomposition.

Built on

Approximating discrete probability distributions with dependence trees
C.K. Chow and C.N. Liu · 1968
Earlier work this paper cites.
Statistical analysis of non-lattice data
Julian Besag · 1975
Earlier work this paper cites.
Information processing in dynamical systems: Foundations of harmony theory
Paul Smolensky · 1986
Earlier work this paper cites.
Parameter inference for imperfectly observed Gibbsian fields
Laurent Younes · 1989
Earlier work this paper cites.
Unsupervised learning of distributions on binary vectors using two layer networks
Yoav Freund and David Haussler · 1992
Earlier work this paper cites.
Connectionist learning of belief networks
Radford M. Neal · 1992
Earlier work this paper cites.
DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM
J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, D. Pallett, N. Dahlgren, and V. Zue · 1993
Earlier work this paper cites.
Mixture density networks
Christopher M. Bishop · 1994
Earlier work this paper cites.
The Helmholtz machine
Peter Dayan, Geoffrey E. Hinton, Radford M. Neal, and Richard S. Zemel · 1995
Earlier work this paper cites.
The wake-sleep algorithm for unsupervised neural networks
Geoffrey E. Hinton, Peter Dayan, Brendan J. Frey, and Radford M. Neal · 1995
Earlier work this paper cites.
Improved Gaussian mixture density estimates using Bayesian penalty terms and network averaging
Dirk Ormoneit and Volker Tresp · 1995
Earlier work this paper cites.
Does the wake-sleep algorithm learn good density estimators?
Brendan J. Frey, Geoffrey E. Hinton, and Peter Dayan · 1996
Earlier work this paper cites.
The EM algorithm for mixtures of factor analyzers
Zoubin Ghahramani and Geoffrey E. Hinton · 1996
Earlier work this paper cites.
Linearly combining density estimators via stacking
Padhraic Smyth and David Wolpert · 1999
Earlier work this paper cites.
Modeling high-dimensional discrete data with multi-layer neural networks
Yoshua Bengio and Samy Bengio · 2000
Earlier work this paper cites.
A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics
D. Martin, C. Fowlkes, D. Tal, and J. Malik · 2001
Earlier work this paper cites.
Training products of experts by minimizing contrastive divergence
Geoffrey E. Hinton · 2002
Earlier work this paper cites.
Estimation of non-normalized statistical models by score matching
Aapo Hyvärinen · 2005
Earlier work this paper cites.
Mixture of factor analyzers Matlab implementation, 2005
Jakob Verbeek · 2005
Earlier work this paper cites.
Exponential family harmoniums with an application to information retrieval
Max Welling, Michal Rosen-Zvi, and Geoffrey E. Hinton · 2005
Earlier work this paper cites.
A fast learning algorithm for deep belief nets
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh · 2006
Earlier work this paper cites.
A kernel method for the two-sample-problem
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex J. Smola · 2007
Earlier work this paper cites.
On the quantitative analysis of deep belief networks
Ruslan Salakhutdinov and Iain Murray · 2008
Earlier work this paper cites.

Similar

Training restricted Boltzmann machines using approximations to the likelihood gradient
Tijmen Tieleman · 2008
Cited alongside, same era.
Extracting and composing robust features with denoising autoencoders
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol · 2008
Cited alongside, same era.
Learning deep architectures for AI
Yoshua Bengio · 2009
Cited alongside, same era.
Learning in Markov random fields using tempered transitions
Ruslan Salakhutdinov · 2009
Cited alongside, same era.
Deep Boltzmann machines
Ruslan Salakhutdinov and Geoffrey E. Hinton · 2009
Cited alongside, same era.
Using fast weights to improve persistent contrastive divergence
Tijmen Tieleman and Geoffrey E. Hinton · 2009
Cited alongside, same era.
Parallel tempering is efficient for learning restricted Boltzmann machines
KyungHyun Cho, Tapani Raiko, and Alexander Ilin · 2010
Cited alongside, same era.
Tempered Markov chain Monte Carlo for training of restricted Boltzmann machine
Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau · 2010
Cited alongside, same era.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Michael Gutmann and Aapo Hyvärinen · 2010
Cited alongside, same era.
Inductive principles for restricted Boltzmann machine learning
Benjamin Marlin, Kevin Swersky, Bo Chen, and Nando de Freitas · 2010
Cited alongside, same era.
Learning deep Boltzmann machines using adaptive MCMC
Ruslan Salakhutdinov · 2010
Cited alongside, same era.
Efficient learning of deep Boltzmann machines
Ruslan Salakhutdinov and Hugo Larochelle · 2010
Cited alongside, same era.

Then

Deep mixtures of factor analysers
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey E. Hinton · 2012
Later among the works it cites.
Natural images, Gaussian mixtures and dead leaves
Daniel Zoran and Yair Weiss · 2012
Later among the works it cites.
UCI machine learning repository, 2013
Kevin Bache and Moshe Lichman · 2013
Later among the works it cites.
Enhanced gradient for training restricted Boltzmann machines
KyungHyun Cho, Tapani Raiko, and Alexander Ilin · 2013
Later among the works it cites.
Improving deep neural networks for LVCSR using rectified linear units and dropout
George E. Dahl, Tara N. Sainath, and Geoffrey E. Hinton · 2013
Later among the works it cites.
RNADE: The real-valued neural autoregressive density-estimator
Benigno Uria, Iain Murray, and Hugo Larochelle · 2013
Later among the works it cites.
Generative adversarial nets
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio · 2014
Later among the works it cites.
Deep autoregressive networks
Karol Gregor, Andriy Mnih, and Daan Wierstra · 2014
Later among the works it cites.
Auto-encoding variational Bayes
Original
Diederik P. Kingma and Max Welling · 2014
Later among the works it cites.
Iterative neural autoregressive distribution estimator (NADE-k)
Tapani Raiko, Li Yao, Kyunghyun Cho, and Yoshua Bengio · 2014
Later among the works it cites.
Stochastic backpropagation and approximate inference in deep generative models
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra · 2014
Later among the works it cites.
Reweighted wake-sleep
Original
Jörg Bornschein and Yoshua Bengio · 2015
Later among the works it cites.
Deep generative image models using a Laplacian pyramid of adversarial networks
Emily L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus · 2015
Later among the works it cites.
MADE: Masked autoencoder for distribution estimation
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle · 2015
Later among the works it cites.
DRAW: a recurrent neural network for image generation
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra · 2015
Later among the works it cites.
Adam: a method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Lei Ba · 2015
Later among the works it cites.
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey E. Hinton · 2015
Later among the works it cites.
Generative moment matching networks
Yujia Li, Kevin Swersky, and Richard S. Zemel · 2015
Later among the works it cites.
Striving for simplicity: the all convolutional net
Original
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller · 2015
Later among the works it cites.
Generative image modeling using spatial lstms
Lucas Theis and Matthias Bethge · 2015
Later among the works it cites.
Connectionist multivariate density-estimation and its application to speech synthesis
Benigno Uria · 2015
Later among the works it cites.
Importance weighted autoencoders
Original
Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse · 2016
Closest in time.
Pixel recurrent neural networks
Original
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…