Fetching the paper…

Reading the bibliography…

2016

Stochastic Backpropagation through Mixture Density Distributions

Graves, Alex

Original
PDF
Post on X

Understand

The ability to backpropagate stochastic gradients through continuous latent distributions has been crucial to the emergence of variational autoencoders and stochastic gradient variational Bayes.

The key ingredient is an unbiased and low-variance way of estimating gradients with respect to distribution parameters from gradients evaluated at distribution samples.
The "reparameterization trick" provides a class of transforms yielding such estimators for many continuous distributions, including the Gaussian and other members of the location-scale family.
However the trick does not readily extend to mixture density models, due to the difficulty of reparameterizing the discrete distribution over mixture weights.

Stochastic Backpropagation through Mixture Density Distributions · Around

Built on

Practical variational inference for neural networks
A. Graves · 2011
Earlier work this paper cites.
Deep autoregressive networks
K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
D. P. Kingma and M. Welling · 2014
Earlier work this paper cites.

Similar

Stochastic backpropagation and approximate inference in deep generative models
Original
D. J. Rezende, S. Mohamed, and D. Wierstra · 2014
Cited alongside, same era.
Weight Uncertainty in Neural Networks
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra · 2015
Cited alongside, same era.

Then

DRAW: A recurrent neural network for image generation
K. Gregor, I. Danihelka, A. Graves, and D. Wierstra · 2015
Later among the works it cites.
Variational dropout and the local reparameterization trick
D. P. Kingma, T. Salimans, and M. Welling · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…