2019

Emerging Convolutions for Generative Normalizing Flows

Hoogeboom, Emiel, Berg, Rianne van den, Welling, Max

Understand

Generative flows are attractive because they admit exact likelihood optimization and efficient image synthesis.

Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images.
We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolutions, which are more flexible since they operate on both channel and spatial axes.
We propose two methods to produce invertible convolutions that have receptive fields identical to standard convolutions: Emerging convolutions are obtained by chaining specific autoregressive convolutions, and periodic convolutions are decoupled in the frequency domain.

Higher Order Statistical Decorrelation without Information Loss
Deco, G. and Brauer, W · 1995
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Density estimation by dual ascent of the log-likelihood
Tabak, E. G., Vanden-Eijnden, E., et al · 2010
Earlier work this paper cites.
High-dimensional probability estimation with deep density models
Original
Rippel, O. and Adams, R. P · 2013
Earlier work this paper cites.
A family of nonparametric density estimation algorithms
Tabak, E. and Turner, C. V · 2013
Earlier work this paper cites.
NICE: Non-linear independent components estimation
Original
Dinh, L., Krueger, D., and Bengio, Y · 2014
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Auto-Encoding Variational Bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D · 2014
Earlier work this paper cites.
Made: Masked autoencoder for distribution estimation
Germain, M., Gregor, K., Murray, I., and Larochelle, H · 2015
Earlier work this paper cites.

Variational Inference with Normalizing Flows
Rezende, D. and Mohamed, S · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015
Cited alongside, same era.
Improved variational inference with inverse autoregressive flow
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M · 2016
Cited alongside, same era.
A note on the evaluation of generative models
Theis, L., van den Oord, A., and Bethge, M · 2016
Cited alongside, same era.
Improving variational auto-encoders using householder flow
Original
Tomczak, J. M. and Welling, M · 2016
Cited alongside, same era.
Conditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al · 2016
Cited alongside, same era.
Pixel recurrent neural networks
Van Oord, A., Kalchbrenner, N., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2017
Cited alongside, same era.
The reversible residual network: Backpropagation without storing activations
Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B · 2017
Cited alongside, same era.
Using transfer learning to detect galaxy mergers
Ackermann, S., Schawinksi, K., Zhang, C., Weigel, A. K., and Turp, M. D
Cited in the paper.

Multiplicative normalizing flows for variational bayesian neural networks
Louizos, C. and Welling, M · 2017
Later among the works it cites.
Masked autoregressive flow for density estimation
Papamakarios, G., Murray, I., and Pavlakou, T · 2017
Later among the works it cites.
A note on the inception score
Barratt, S. and Sharma, R · 2018
Later among the works it cites.
Neural ordinary differential equations
Chen, T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K · 2018
Later among the works it cites.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Grathwohl, W., Chen, R. T., Betterncourt, J., Sutskever, I., and Duvenaud, D · 2018
Later among the works it cites.
Neural autoregressive flows
Original
Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A · 2018
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P · 2018
Later among the works it cites.
Training Glow with Constant Memory Cost
Li, X. and Grathwohl, W · 2018
Later among the works it cites.
Sylvester normalizing flows for variational inference
Original
van den Berg, R., Hasenclever, L., Tomczak, J. M., and Welling, M · 2018
Later among the works it cites.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…