2020

SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows

Nielsen, Didrik, Jaini, Priyank, Hoogeboom, Emiel et al.

Understand

Normalizing flows and variational autoencoders are powerful generative models that can represent complicated density functions.

However, they both impose constraints on the models: Normalizing flows use bijective transformations to model densities whereas VAEs learn stochastic transformations that are non-invertible and thus typically do not provide tractable estimates of the marginal likelihood.
In this paper, we introduce SurVAE Flows: A modular framework of composable transformations that encompasses VAEs and normalizing flows.
SurVAE Flows bridge the gap between normalizing flows and VAEs with surjective transformations, wherein the transformations are deterministic in one direction -- thereby allowing exact likelihood computation, and stochastic in the reverse direction -- hence providing a lower bound on the corresponding likelihood.

Built on

A rad approach to deep mixture models
Original
Dinh, L., Sohl-Dickstein, J., Pascanu, R., and Larochelle, H. (2019) · 1903
Earlier work this paper cites.
Transport monte carlo
Original
Duan, L. L. (2019) · 1907
Earlier work this paper cites.
Tails of triangular flows
Original
Jaini, P., Kobyzev, I., Brubaker, M., and Yu, Y. (2019a) · 1907
Earlier work this paper cites.
Localised generative flows
Original
Cornish, R., Caterini, A. L., Deligiannidis, G., and Doucet, A. (2019) · 1909
Earlier work this paper cites.
Probabilistic principal component analysis
Tipping, M. E. and Bishop, C. M. (1999) · 1999
Earlier work this paper cites.
Vflow: More expressive generative flows with variational data augmentation
Original
Chen, J., Lu, C., Chenli, B., Zhu, J., and Tian, T. (2020) · 2002
Earlier work this paper cites.
Augmented normalizing flows: Bridging the gap between generative flows and latent variable models
Original
Huang, C., Dinh, L., and Courville, A. C. (2020) · 2002
Earlier work this paper cites.
Denoising diffusion probabilistic models
Original
Ho, J., Jain, A., and Abbeel, P. (2020) · 2006
Earlier work this paper cites.
The convolution exponential and generalized sylvester flows
Original
Hoogeboom, E., Satorras, V. G., Tomczak, J. M., and Welling, M. (2020) · 2006
Earlier work this paper cites.
Density estimation by dual ascent of the log-likelihood
Tabak, E. G. and Vanden-Eijnden, E. (2010) · 2010
Earlier work this paper cites.
A family of nonparametric density estimation algorithms
Tabak, E. G. and Turner, C. V. (2013) · 2013
Earlier work this paper cites.
Rnade: The real-valued neural autoregressive density-estimator
Uria, B., Murray, I., and Larochelle, H. (2013) · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M. (2014) · 2014
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014) · 2014
Earlier work this paper cites.

Similar

A deep and tractable density estimator
Uria, B., Murray, I., and Larochelle, H. (2014) · 2014
Cited alongside, same era.
NICE: Non-linear independent components estimation
Dinh, L., Krueger, D., and Bengio, Y. (2015) · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J. (2015) · 2015
Cited alongside, same era.
Locally-connected transformations for deep gmms
Oord, A. v. d. and Dambre, J. (2015) · 2015
Cited alongside, same era.
Variational inference with normalizing flows
Rezende, D. J. and Mohamed, S. (2015) · 2015
Cited alongside, same era.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. (2015) · 2015
Cited alongside, same era.
Importance weighted autoencoders
Burda, Y., Grosse, R. B., and Salakhutdinov, R. (2016) · 2016
Cited alongside, same era.
Improving variational autoencoders with inverse autoregressive flow
Kingma, D. P., Salimans, T., Józefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016) · 2016
Cited alongside, same era.
A note on the evaluation of generative models
Theis, L., van den Oord, A., and Bethge, M. (2016) · 2016
Cited alongside, same era.
Tensorflow distributions
Original
Dillon, J. V., Langmore, I., Tran, D., Brevdo, E., Vasudevan, S., Moore, D., Patton, B., Alemi, A., Hoffman, M. D., and Saurous, R. A. (2017) · 2017
Cited alongside, same era.
Density estimation using real NVP
Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017) · 2017
Cited alongside, same era.
Towards a neural statistician
Edwards, H. and Storkey, A. J. (2017) · 2017
Cited alongside, same era.

Then

Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017) · 2017
Later among the works it cites.
Pyro: Deep Universal Probabilistic Programming
Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., and Goodman, N. D. (2018) · 2018
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P. (2018) · 2018
Later among the works it cites.
BRUNO: A deep recurrent model for exchangeable data
Korshunova, I., Degrave, J., Huszar, F., Gal, Y., Gretton, A., and Dambre, J. (2018) · 2018
Later among the works it cites.
Autoregressive quantile networks for generative modeling
Ostrovski, G., Dabney, W., and Munos, R. (2018) · 2018
Later among the works it cites.
Sylvester normalizing flows for variational inference
van den Berg, R., Hasenclever, L., Tomczak, J. M., and Welling, M. (2018) · 2018
Later among the works it cites.
Augmented neural odes
Dupont, E., Doucet, A., and Teh, Y. W. (2019) · 2019
Later among the works it cites.
Neural spline flows
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019) · 2019
Later among the works it cites.
Flow++: Improving flow-based generative models with variational dequantization and architecture design
Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. (2019) · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019) · 2019
Later among the works it cites.
Discrete flows: Invertible generative models of discrete data
Tran, D., Vafa, K., Agrawal, K. K., Dinh, L., and Poole, B. (2019) · 2019
Later among the works it cites.
Exchangeable generative models with flow scans
Bender, C. M., O’Connor, K., Li, Y., Garcia, J. J., Oliva, J., and Zaheer, M. (2020) · 2020
Closest in time.
Stochastic normalizing flows
Wu, H., Köhler, J., and Noé, F. (2020) · 2020
Closest in time.
Neural autoregressive flows
Huang, C., Krueger, D., Lacoste, A., and Courville, A. C. (2018) · 2092
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…