2016

R\'enyi Divergence Variational Inference

Li, Yingzhen, Turner, Richard E.

Understand

This paper introduces the variational R\'enyi bound (VR) that extends traditional variational inference to R\'enyi's alpha-divergences.

This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is controlled by the value of alpha that parametrises the divergence.
The reparameterization trick, Monte Carlo approximation and stochastic optimisation methods are deployed to obtain a tractable and unified framework for optimisation.
We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework.

Built on

A. Rényi, “On measures of entropy and information,” Fourth Berkeley symposium on mathematical statistics and probability
1961
Earlier work this paper cites.
New York: Springer, 1985
S.-i. Amari, Differential-Geometrical Methods in Statistic · 1985
Earlier work this paper cites.
C. Tsallis, “Possible generalization of Boltzmann-Gibbs statistics,” Journal of statistical physics
1988
Earlier work this paper cites.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction to variational methods for graphical models,” Machine learning
1999
Earlier work this paper cites.
T. Minka, “Expectation propagation for approximate Bayesian inference,” in Conference on Uncertainty in Artificial Intelligence (UAI)
2001
Earlier work this paper cites.
PhD thesis, University College London, 2003
M. J. Beal, Variational algorithms for approximate Bayesian inference · 2003
Earlier work this paper cites.
T. Minka, “Power EP,” Tech. Rep. MSR-TR-2004-149, Microsoft Research, 2004
2004
Earlier work this paper cites.
M. Opper and O. Winther, “Expectation consistent approximate inference,” The Journal of Machine Learning Research
2005
Earlier work this paper cites.
T. Minka, “Divergence measures and message passing,” tech. rep., Microsoft Research, 2005
2005
Earlier work this paper cites.
MIT press, Cambridge, MA, 2007
P. Grünwald, Minimum Description Length Principle · 2007
Earlier work this paper cites.

Similar

R. E. Turner and M. Sahani, “Two problems with variational expectation maximisation for time-series models,” in Bayesian Time series models
2011
Cited alongside, same era.
J. Paisley, D. Blei, and M. Jordan, “Variational Bayesian inference with stochastic search,” in Proceedings of The 29th International Conference on Machine Learning (ICML)
2012
Cited alongside, same era.
M. D. Hoffman, D. M. Blei, C. Wang, and J. W. Paisley, “Stochastic variational inference,” Journal of Machine Learning Research
2013
Cited alongside, same era.
T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, and M. I. Jordan, “Streaming variational Bayes,” in Advances in Neural Information Processing Systems (NIPS)
2013
Cited alongside, same era.
T. Salimans and D. A. Knowles, “Fixed-form variational posterior approximation through stochastic linear regression,” Bayesian Analysis
2013
Cited alongside, same era.
A. Gelman, A. Vehtari, P. Jylänki, C. Robert, N. Chopin, and J. P. Cunningham, “Expectation propagation as a way of life,” arXiv:1412.4869
Original
2014
Cited alongside, same era.
M. Xu, B. Lakshminarayanan, Y. W. Teh, J. Zhu, and B. Zhang, “Distributed Bayesian posterior sampling via moment sharing,” in Advances in Neural Information Processing Systems (NIPS)
2014
Cited alongside, same era.
R. Ranganath, S. Gerrish, and D. M. Blei, “Black box variational inference,” in Proceedings of the 17th International Conference on Artificial Intelligence and Statistics (AISTATS)
2014
Cited alongside, same era.
D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in International Conference on Learning Representations (ICLR)
2014
Cited alongside, same era.
D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” in Proceedings of The 30th International Conference on Machine Learning (ICML)
2014
Cited alongside, same era.

Then

T. Van Erven and P. Harremoës, “Rényi divergence and Kullback-Leibler divergence,” Information Theory, IEEE Transactions on
2014
Later among the works it cites.
Y. Li, J. M. Hernández-Lobato, and R. E. Turner, “Stochastic expectation propagation,” in Advances in Neural Information Processing Systems (NIPS)
2015
Later among the works it cites.
G. Dehaene and S. Barthelmé, “Expectation propagation in the large-data limit,” arXiv:1503.08060
Original
2015
Later among the works it cites.
A. Kucukelbir, R. Ranganath, A. Gelman, and D. M. Blei, “Automatic variational inference in Stan,” in Advances in Neural Information Processing Systems (NIPS)
2015
Later among the works it cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR)
2015
Later among the works it cites.
J. M. Hernández-Lobato, Y. Li, M. Rowland, D. Hernández-Lobato, T. Bui, and R. E. Turner, “Black-box α \alpha -divergence minimization,” in Proceedings of The 33rd International Conference on Machine Learning (ICML)
2016
Closest in time.
Y. Burda, R. Grosse, and R. Salakhutdinov, “Importance weighted autoencoders,” in International Conference on Learning Representations (ICLR)
2016
Closest in time.
T. D. Bui, D. Hernández-Lobato, Y. Li, J. M. Hernández-Lobato, and R. E. Turner, “Deep gaussian processes for regression using approximate expectation propagation,” in Proceedings of The 33rd International Conference on Machine Learning (ICML)
2016
Closest in time.
S. Depeweg, J. M. Hernández-Lobato, F. Doshi-Velez, and S. Udluft, “Learning and policy search in stochastic dynamical systems with bayesian neural networks,” arXiv preprint arXiv:1605.07127
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…