2017

Advances in Variational Inference

Zhang, Cheng, Butepage, Judith, Kjellstrom, Hedvig et al.

Understand

Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models.

These models are usually intractable and thus require approximate inference.
Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem.
This approach has been successfully used in various models and large-scale applications.

Built on

On information and sufficiency
S. Kullback and R. A. Leibler · 1951
Earlier work this paper cites.
A stochastic approximation method
H. Robbins and S. Monro · 1951
Earlier work this paper cites.
Eine informationstheoretische ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten
I. Csiszár · 1964
Earlier work this paper cites.
A general class of coefficients of divergence of one distribution from another
S. M. Ali and S. D. Silvey · 1966
Earlier work this paper cites.
A bound for the error in the normal approximation to the distribution of a sum of dependent random variables
C. Stein · 1972
Earlier work this paper cites.
Options: A monte carlo approach
P.P. Boyle · 1977
Earlier work this paper cites.
Solution of ’solvable model of a spin glass’
D.J. Thouless, P. W. Anderson, and R. G. Palmer · 1977
Earlier work this paper cites.
Convergence condition of the TAP equation for the infinite-ranged ising spin glass model
T Plefka · 1982
Earlier work this paper cites.
Differential-geometrical methods in statistics
S.I. Amari · 1985
Earlier work this paper cites.
Memoir on the probability of the causes of events
P. S. Laplace · 1986
Earlier work this paper cites.
Accurate approximations for posterior moments and marginal densities
L. Tierney and J.B. Kadane · 1986
Earlier work this paper cites.
A mean field theory learning algorithm for neural networks
C. Peterson and J. R. Anderson · 1987
Earlier work this paper cites.
Statistical field theory
G. Parisi · 1988
Earlier work this paper cites.
Spin glass theory and beyond
M. Mézard, G. Parisi, and M. A. Virasoro · 1990
Earlier work this paper cites.
A deterministic annealing approach to clustering
K. Rose, E. Gurewitz, and G. Fox · 1990
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
R. J. Williams · 1992
Earlier work this paper cites.
Probabilistic inference using markov chain monte carlo methods
R. M. Neal · 1993
Earlier work this paper cites.
Laplace’s method approximations for probabilistic inferencein belief networks with continuous variables
A. Azevedo-Filho and R.D. Shachter · 1994
Earlier work this paper cites.
The helmholtz machine
P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel · 1995
Earlier work this paper cites.
Introduction to mathematical statistics.(5th edition)
R. V. Hogg and A. T. Craig · 1995
Earlier work this paper cites.
Information geometric measurements of generalisation
H. Zhu and R. Rohwer · 1995
Earlier work this paper cites.
Hidden Markov Models
S. R. Eddy · 1996
Earlier work this paper cites.
Fast learning by bounding likelihoods in sigmoid type belief networks
T. S. Jaakkol, L. K. Saul, and M. I. Jordan · 1996
Earlier work this paper cites.
A mean field algorithm for bayes learning in large feed-forward neural networks
M. Opper and O. Winther · 1996
Earlier work this paper cites.
Mean field theory for sigmoid belief networks
L. K. Saul, T. Jaakkola, and M. I. Jordan · 1996
Earlier work this paper cites.
Integral probability metrics and their generating classes of functions
A. Müller · 1997
Earlier work this paper cites.
Natural gradient works efficiently in learning
S.I. Amari · 1998
Earlier work this paper cites.
Improving the mean field approximation via the use of mixture distributions
T. S. Jaakkola and M. I. Jordan · 1998
Earlier work this paper cites.
Reinforcement learning: An introduction
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Estimation of third-order correlations within mean field approximation
T. Tanaka · 1998
Earlier work this paper cites.
An introduction to variational methods for graphical models
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul · 1999
Earlier work this paper cites.
Loopy belief propagation for approximate inference: An empirical study
K. P. Murphy, Y. Weiss, and M. I. Jordan · 1999
Earlier work this paper cites.
A theory of mean field approximation
T. Tanaka · 1999
Earlier work this paper cites.
Information geometry of mean-field approximation
T. Tanaka · 2000
Earlier work this paper cites.
The information bottleneck method
N. Tishby, F. C. Pereira, and W. Bialek · 2000
Earlier work this paper cites.
Second order approximations for probability models
H. J. Kappen and W. Wiegerinck · 2001
Earlier work this paper cites.
Expectation propagation for approximate bayesian inference
T. P. Minka · 2001
Earlier work this paper cites.
Advanced mean field methods: Theory and practice
M. Opper and D. Saad · 2001
Earlier work this paper cites.
Tractable approximations for probabilistic models: The adaptive thouless-anderson-palmer mean field approach
M. Opper and O. Winther · 2001
Earlier work this paper cites.
Online model selection based on the variational Bayes
M. Sato · 2001
Earlier work this paper cites.
The IM algorithm: a variational approach to information maximization
D. Barber and F. Agakov · 2003
Earlier work this paper cites.
Online variational bayesian learning
A. Honkela and H. Valpola · 2003
Earlier work this paper cites.
Power EP
T. Minka · 2004
Earlier work this paper cites.
Divergence measures and message passing
T. P. Minka · 2005
Earlier work this paper cites.
Expectation propagation for exponential families
M. Seeger · 2005
Earlier work this paper cites.
A new class of upper bounds on the log partition function
M. J. Wainwright, T. S. Jaakkola, and A. S. Willsky · 2005
Earlier work this paper cites.
Variational message passing
J. Winn and C. M. Bishop · 2005
Earlier work this paper cites.
Pattern recognition and machine learning
C. M. Bishop · 2006
Earlier work this paper cites.
Correlated topic models
D. M. Blei and J. D. Lafferty · 2006
Earlier work this paper cites.
Dynamic topic models
D. M. Blei and J. D. Lafferty · 2006
Earlier work this paper cites.
Fast variational inference for gaussian process models through KL-correction
N. King and N. Lawrence · 2006
Earlier work this paper cites.
Simulation
S. M. Ross · 2006
Earlier work this paper cites.
Sparse gaussian processes using pseudo-inputs
E. Snelson and Z. Ghahramani · 2006
Earlier work this paper cites.
A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation
Y. W. Teh, D. Newman, and M. Welling · 2006
Earlier work this paper cites.
Natural conjugate gradient in variational inference
A. Honkela, M. Tornio, T. Raiko, and J. Karhunen · 2007
Earlier work this paper cites.
Collapsed variational dirichlet process mixture models
K. Kurihara, M. Welling, and Y. W. Teh · 2007
Earlier work this paper cites.
Variational bayes solution of linear neural networks and its generalization performance
S. Nakajima and S. Watanabe · 2007
Earlier work this paper cites.
Parallelized variational em for latent dirichlet allocation: An experimental evaluation of speed and scalability
R. Nallapati, W. Cohen, and J. Lafferty · 2007
Earlier work this paper cites.
Parameter estimation for text analysis, 2008
G. Heinrich · 2008
Earlier work this paper cites.
Fast collapsed gibbs sampling for latent dirichlet allocation
I. Porteous, D. Newman, A. Ihler, A. Asuncion, P. Smyth, and M. Welling · 2008
Earlier work this paper cites.
Latent-space variational bayes
J. Sung, Z. Ghahramani, and S. Y. Bang · 2008
Earlier work this paper cites.
Graphical models, exponential families, and variational inference
M. J. Wainwright and M. I. Jordan · 2008
Earlier work this paper cites.
α \alpha -divergence is unique, belonging to both f f -divergence and bregman divergence classes
S.I. Amari · 2009
Earlier work this paper cites.
On integral probability metrics, \ \backslash phi-divergences and binary classification
Original
B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. RG. Lanckriet · 2009
Earlier work this paper cites.
Variational learning of inducing variables in sparse gaussian processes
M. K. Titsias · 2009
Earlier work this paper cites.
Simultaneous image classification and annotation
C. Wang, D. Blei, and L. Fei-Fei · 2009
Earlier work this paper cites.
Large-scale machine learning with stochastic gradient descent
L. Bottou · 2010
Earlier work this paper cites.
Online learning for Latent Dirichlet Allocation
M. Hoffman, F. R. Bach, and D. M. Blei · 2010
Earlier work this paper cites.
Approximate riemannian conjugate gradient learning for fixed-form variational bayes
A. Honkela, T. Raiko, M. Kuusela, M. Tornio, and J. Karhunen · 2010
Earlier work this paper cites.
Handbook of markov chain monte carlo
S. Brooks, A. Gelman, G. Jones, and X.L Meng · 2011
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
J. Duchi, E. Hazan, and Y. Singer · 2011
Earlier work this paper cites.
Approximate inference in graphical models
P. Hennig · 2011
Earlier work this paper cites.
Non-conjugate variational message passing for multinomial and binary regression
D. A. Knowles and T. P. Minka · 2011
Earlier work this paper cites.
Variational heteroscedastic gaussian process regression
M.K. Titsias and M. Lázaro-Gredilla · 2011
Earlier work this paper cites.
Online variational inference for the hierarchical dirichlet process
C. Wang, J. Paisley, and D. M. Blei · 2011
Earlier work this paper cites.
Bayesian learning via stochastic gradient langevin dynamics
M. Welling and Y. W. Teh · 2011
Earlier work this paper cites.

Similar

Bayesian posterior sampling via stochastic gradient fisher scoring
S. Ahn, A. Korattikara, and M. Welling · 2012
Cited alongside, same era.
Soamploample size selection in optimization methods for machine learning
R.H. Byrd, G.M. Chin, J. Nocedal, and Y.C. Wu · 2012
Cited alongside, same era.
Asymptotic methods in statistical decision theory
L. Le Cam · 2012
Cited alongside, same era.
Hybrid deterministic-stochastic methods for data fitting
M. P. Friedlander and M. Schmidt · 2012
Cited alongside, same era.
Nonparametric variational inference
S. J. Gershman, M. D. Hoffman, and D. M. Blei · 2012
Cited alongside, same era.
Scalable inference of overlapping communities
P. K. Gopalan, S. Gerrish, M. Freedman, D. M. Blei, and D. M. Mimno · 2012
Cited alongside, same era.
Fast variational inference in the conjugate exponential family
J Hensman, M Rattray, and N. D Lawrence · 2012
Cited alongside, same era.
Overlapping mixtures of gaussian processes for the data association problem
M. Lázaro-Gredilla, S. Van Vaerenbergh, and N. D. Lawrence · 2012
Cited alongside, same era.
Bayesian learning for neural networks
R. M. Neal · 2012
Cited alongside, same era.
Variational bayesian inference with stochastic search
J. Paisley, D. M. Blei, and M. I. Jordan · 2012
Cited alongside, same era.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
T. Tieleman and G. Hinton · 2012
Cited alongside, same era.
Adadelta: an adaptive learning rate method
Original
M.D. Zeiler · 2012
Cited alongside, same era.

Then

Black-Box Alpha Divergence Minimization
Y.Z Li, M. Rowland, T. Bui, D. Hernandez-Lobato, and R. Turner · 2016
Later among the works it cites.
Two methods for wild variational inference
Original
Q. Liu and Y. Feng · 2016
Later among the works it cites.
A kernelized stein discrepancy for goodness-of-fit tests
Q. Liu, J. Lee, and M. Jordan · 2016
Later among the works it cites.
Stein variational gradient descent: A general purpose bayesian inference algorithm
Q. Liu and D. Wang · 2016
Later among the works it cites.
A Variational Analysis of Stochastic Gradient Algorithms
S. Mandt, M. D. Hoffman, and D. M. Blei · 2016
Later among the works it cites.
Variational Tempering
S. Mandt, J. McInerney, F. Abrol, R. Ranganath, and Blei · 2016
Later among the works it cites.
Neural variational inference for text processing
Y. Miao, L. Yu, and P. Blunsom · 2016
Later among the works it cites.
Learning in implicit generative models
Original
S. Mohamed and B. Lakshminarayanan · 2016
Later among the works it cites.
Operator variational inference
R. Ranganath, D. Tran, J. Altosaar, and D. M. Blei · 2016
Later among the works it cites.
Hierarchical variational models
R. Ranganath, D. Tran, and D. M. Blei · 2016
Later among the works it cites.
The generalized reparameterization gradient
F. J.R. Ruiz, M. K. Titsias, and D. M. Blei · 2016
Later among the works it cites.
Overdispersed black-box variational inference
F. J.R. Ruiz, M. K. Titsias, and D. M. Blei · 2016
Later among the works it cites.
How to train deep variational autoencoders and probabilistic ladder networks
C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, and O. Winther · 2016
Later among the works it cites.
Design and implementation of probabilistic programming language anglican
Original
D. Tolpin, J. W. van de Meent, H. Yang, and F. Wood · 2016
Later among the works it cites.
Edward: A library for probabilistic modeling, inference, and criticism
Original
D. Tran, A. Kucukelbir, A. B. Dieng, M. Rudolph, D. Liang, and D. M. Blei · 2016
Later among the works it cites.
The Variational Gaussian Process
D. Tran, R. Ranganath, and D. M. Blei · 2016
Later among the works it cites.
Learning to draw samples: With application to amortized MLE for generative adversarial learning
Original
D. Wang and Q. Liu · 2016
Later among the works it cites.
Structured Representation Using Latent Variable Models
C. Zhang · 2016
Later among the works it cites.
Online spike-and-slab inference with stochastic expectation propagation
S.D. Zhe, K.C. Lee, K. Zhang, and J. Neville · 2016
Later among the works it cites.
Deep variational information bottleneck
A. Alemi, I. Fischer, J. Dillon, and K. Murphy · 2017
Closest in time.
Coupling adaptive batch sizes with learning rates
L. Balles, J. Romero, and P. Hennig · 2017
Closest in time.
Dynamic word embeddings
R. Bamler and S. Mandt · 2017
Closest in time.
Structured black box variational inference for latent time series models
R. Bamler and S. Mandt · 2017
Closest in time.
Variational inference: A review for statisticians
D. M. Blei, A. Kucukelbir, and J. D. McAuliffe · 2017
Closest in time.
Variational lossy autoencoder
X. Chen, D. P. Kingma, T. Salimans, Y. Dua, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel · 2017
Closest in time.
Reinterpreting importance-weighted autoencoders
C. Cremer, Q. Morris, and D. Duvenaud · 2017
Closest in time.
Automated inference using adaptive batch sizes
S. De, A. Yadav, D. Jacobs, and T. Goldstein · 2017
Closest in time.
Factorized variational autoencoders for modeling audience reactions to movies
Z.W. Deng, R. Navarathna, P. Carr, S. Mandt, Y.S. Yue, I. Matthews, and G. Mori · 2017
Closest in time.
Variational inference via chi-upper bound minimization
A. B. Dieng, D. Tran, R. Ranganath, J. Paisley, and D. M. Blei · 2017
Closest in time.
Density estimation using real nvp
L. Dinh, J. Sohl-Dickstein, and S. Bengio · 2017
Closest in time.
CPSG-MCMC: Clustering-based preprocessing method for stochastic gradient mcmc
T.F Fu and Z.H. Zhang · 2017
Closest in time.
PixelVAE: A latent variable model for natural images
I. Gulrajani, K. Kumar, F. Ahmed, A. A. Taiga, F. V. Visin, D. Vazquez, and A. Courville · 2017
Closest in time.
Stein variational adaptive importance sampling
J. Han and Q. Liu · 2017
Closest in time.
Variational inference using implicit distributions
Original
F. Huszár · 2017
Closest in time.
Categorical reparameterization with gumbel-softmax
E. Jang, S.X. Gu, and B. Poole · 2017
Closest in time.
Approximate inference with amortised MCMC
Original
Y.Z. Li, R. E. Turner, and Q. Liu · 2017
Closest in time.
Stein variational policy gradient
Y. Liu, P. Ramachandran, Q. Liu, and J. Peng · 2017
Closest in time.
The concrete distribution: A continuous relaxation of discrete random variables, 2017
C. J. Maddison, A. Mnih, and Y. W. Teh · 2017
Closest in time.
Stochastic gradient descent as approximate bayesian inference
S. Mandt, M. D. Hoffman, and D. M. Blei · 2017
Closest in time.
Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks
L. Mescheder, S. Nowozin, and A. Geiger · 2017
Closest in time.
The numerics of gans
L. Mescheder, S. Nowozin, and A. Geiger · 2017
Closest in time.
Reducing reparameterization gradient variance
A. C. Miller, N. J. Foti, A. D’Amour, and R. P. Adams · 2017
Closest in time.
Variational boosting: Iteratively refining posterior approximations
A.C. Miller, N. Foti, and R. P. Adams · 2017
Closest in time.
Reparameterization gradients through acceptance-rejection sampling algorithms
C. Naesseth, F. Ruiz, S. Linderman, and D. M. Blei · 2017
Closest in time.
Stick-breaking variational autoencoders
E. Nalisnick and P. Smyth · 2017
Closest in time.
Control functionals for monte carlo integration
C.J. Oates, M. Girolami, and N. Chopin · 2017
Closest in time.
Faster coordinate descent via adaptive importance sampling
D Perekrestenko, V Cevher, and M Jaggi · 2017
Closest in time.
Sticking the landing: An asymptotically zero-variance gradient estimator for variational inference
G. Roeder, Y. Wu, and D. Duvenaud · 2017
Closest in time.
Discrete variational autoencoders
J. T. Rolfe · 2017
Closest in time.
Multimodal prediction and personalization of photo edits with deep generative models
Original
A. Saeedi, M. D. Hoffman, S. J. DiVerdi, A. Ghandeharioun, M. J. Johnson, and R. P. Adams · 2017
Closest in time.
Zhusuan: A library for bayesian deep learning
Original
J. Shi, J. Chen, J. Zhu, S. Sun, Y. Luo, Yihong Gu, and Yuhao Zhou · 2017
Closest in time.
Autoencoding variational inference for topic models
A. Srivastava and C. Sutton · 2017
Closest in time.
Stochastic variational inference for large-scale discrete choice models using adaptive batch sizes
L. SL. Tan · 2017
Closest in time.
Learning model reparametrizations: Implicit variational inference by fitting mcmc distributions
Original
M. K. Titsias · 2017
Closest in time.
Hierarchical implicit models and likelihood-free variational inference
Original
D. Tran, R. Ranganath, and D. M. Blei · 2017
Closest in time.
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
G. Tucker, A. Mnih, C.J. Maddison, J. Lawson, and J. Sohl-Dickstein · 2017
Closest in time.
Variational policy for guiding point processes
Y. Wang, G. Williams, E. Theodorou, and L. Song · 2017
Closest in time.
Robust Probabilistic Modeling with Bayesian Data Reweighting
Y. X. Wang, A. Kucukelbir, and D. M. Blei · 2017
Closest in time.
Perturbative black box variational inference
C. Zhang, R. Bamler, M. Opper, and S. Mandt · 2017
Closest in time.
Determinantal point processes for mini-batch diversification
C. Zhang, H. Kjellstrom, and S. Mandt · 2017
Closest in time.
Towards deeper understanding of variational autoencoding models
Original
S. Zhao, J. Song, and S. Ermon · 2017
Closest in time.
Efficient gradient-free variational inference using policy search
O. Arenz, M.J. Zhong, and G. Neumann · 2018
Closest in time.
Quasi-Monte Carlo Variational Inference
A. Buchholz, F. Wenzel, and S. Mandt · 2018
Closest in time.
Inference suboptimality in variational autoencoders
C. Cremer, X Li, and D. Duvenaud · 2018
Closest in time.
Reinforcement learning and control as probabilistic inference: Tutorial and review
Original
S. Levine · 2018
Closest in time.
Approximate Inference: New Visions
Y.Z. Li · 2018
Closest in time.
Variational message passing with structured inference networks
Original
Wu. Lin, N. Hubacher, and M.E. Khan · 2018
Closest in time.
Iterative amortized inference
Joseph Marino, Yisong Yue, and Stephan Mandt · 2018
Closest in time.
Fast and scalable bayesian deep learning by weight-perturbation in adam
Original
M.E.¬Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava · 2018
Closest in time.
Debiasing evidence approximations: On importance-weighted autoencoders and jackknife variational inference
S. Nowozin · 2018
Closest in time.
On Nesting Monte Carlo Estimators
T. Rainforth, R. Cornish, H. Yang, A. Warrington, and F. Wood · 2018
Closest in time.
Tighter variational bounds are not necessarily better
T. Rainforth, A. R. Kosiorek, T. A. Le, C. J. Maddison, M. Igl, F. Wood, and Y. W. Teh · 2018
Closest in time.
Frequentist consistency of variational bayes
Yixin Wang and David M Blei · 2018
Closest in time.
Semi-implicit variational inference
M. Yin and M. Zhou · 2018
Closest in time.
Active mini-batch sampling using repulsive point processes
Original
C. Zhang, C. Öztireli, S. Mandt, and G. Salvi · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…