2020

Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities

Köhler, Jonas, Klein, Leon, Noé, Frank

Understand

Normalizing flows are exact-likelihood generative neural networks which approximately transform samples from a simple prior distribution to samples of the probability distribution of interest.

Recent work showed that such generative models can be utilized in statistical mechanics to sample equilibrium states of many-body systems in physics and chemistry.
To scale and generalize these results, it is essential that the natural symmetries in the probability density -- in physics defined by the invariances of the target potential -- are built into the flow.
We provide a theoretical sufficient criterion showing that the distribution generated by \textit{equivariant} normalizing flows is invariant with respect to these symmetries by design.

Built on

Efficient estimation of free energy differences from monte carlo data
Bennett, C. H · 1976
Earlier work this paper cites.
Nonphysical Sampling Distributions in Monte Carlo Free-Energy Estimation: Umbrella Sampling
Torrie, G. M. and Valleau, J. P · 1977
Earlier work this paper cites.
A family of embedded runge-kutta formulae
Dormand, J. R. and Prince, P. J · 1980
Earlier work this paper cites.
A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines
Hutchinson, M. F · 1989
Earlier work this paper cites.
Global optimization by basin-hopping and the lowest energy structures of lennard-jones clusters containing up to 100 atoms
Wales, D. J. and Doye, J. P. K · 1997
Earlier work this paper cites.
A computational fluid mechanics solution to the monge-kantorovich mass transfer problem
Benamou, J.-D. and Brenier, Y · 2000
Earlier work this paper cites.
Polar factorization of maps on riemannian manifolds
McCann, R. J · 2001
Earlier work this paper cites.
Escaping free energy minima
Laio, A. and Parrinello, M · 2002
Earlier work this paper cites.
Density estimation by dual ascent of the log-likelihood
Tabak, E. G., Vanden-Eijnden, E., et al · 2010
Earlier work this paper cites.
A family of nonparametric density estimation algorithms
Tabak, E. G. and Turner, C. V · 2013
Earlier work this paper cites.
Nice: Non-linear independent components estimation
Original
Dinh, L., Krueger, D., and Bengio, Y · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Made: Masked autoencoder for distribution estimation
Germain, M., Gregor, K., Murray, I., and Larochelle, H · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Original
Rezende, D. J. and Mohamed, S · 2015
Earlier work this paper cites.
Density estimation using real nvp
Original
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2016
Earlier work this paper cites.

Similar

Improving variational auto-encoders using householder flow
Original
Tomczak, J. M. and Welling, M · 2016
Cited alongside, same era.
Multiensemble markov models of molecular thermodynamics and kinetics
Wu, H., Paul, F., Wehmeyer, C., and Noé, F · 2016
Cited alongside, same era.
Continuous-time flows for efficient inference and density estimation
Original
Chen, C., Li, C., Chen, L., Wang, W., Pu, Y., and Carin, L · 2017
Cited alongside, same era.
Fixing weight decay regularization in adam
Original
Loshchilov, I. and Hutter, F · 2017
Cited alongside, same era.
Multiplicative normalizing flows for variational bayesian neural networks
Louizos, C. and Welling, M · 2017
Cited alongside, same era.
Parallel wavenet: Fast high-fidelity speech synthesis
Original
Oord, A. v. d., Li, Y., Babuschkin, I., Simonyan, K., Vinyals, O., Kavukcuoglu, K., Driessche, G. v. d., Lockhart, E., Cobo, L. C., Stimberg, F., et al · 2017
Cited alongside, same era.
Masked autoregressive flow for density estimation
Papamakarios, G., Pavlakou, T., and Murray, I · 2017
Cited alongside, same era.
Schnet: A continuous-filter convolutional neural network for modeling quantum interactions
Schütt, K., Kindermans, P.-J., Felix, H. E. S., Chmiela, S., Tkatchenko, A., and Müller, K.-R · 2017
Cited alongside, same era.
Invertible residual networks
Original
Behrmann, J., Grathwohl, W., Chen, R. T., Duvenaud, D., and Jacobsen, J.-H · 2018
Cited alongside, same era.
Sylvester normalizing flows for variational inference
Original
Berg, R. v. d., Hasenclever, L., Tomczak, J. M., and Welling, M · 2018
Cited alongside, same era.
Neural ordinary differential equations
Chen, T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K · 2018
Cited alongside, same era.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., and Duvenaud, D · 2018
Cited alongside, same era.

Then

Fixing weight decay regularization in adam, corr, abs/1711.05101
Original
Loshchilov, I. and Hutter, F · 2018
Later among the works it cites.
Neural importance sampling
Original
Müller, T., McWilliams, B., Rousselle, F., Gross, M., and Novák, J · 2018
Later among the works it cites.
Monge-ampere flow for generative modeling
Original
Zhang, L., Wang, L., et al · 2018
Later among the works it cites.
Flow-based generative models for markov chain monte carlo in lattice field theory
Albergo, M. S., Kanwar, G., and Shanahan, P. E · 2019
Later among the works it cites.
Neural networks with cheap differential operators
Chen, T. Q. and Duvenaud, D. K · 2019
Later among the works it cites.
Residual flows for invertible generative modeling
Chen, T. Q., Behrmann, J., Duvenaud, D. K., and Jacobsen, J.-H · 2019
Later among the works it cites.
Block neural autoregressive flow
Original
De Cao, N., Titov, I., and Aziz, W · 2019
Later among the works it cites.
Neural spline flows
Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G · 2019
Later among the works it cites.
Anode: Unconditionally accurate memory-efficient gradients for neural odes
Original
Gholami, A., Keutzer, K., and Biros, G · 2019
Later among the works it cites.
Equivariant flows: sampling configurations for multi-body systems with symmetric energies
Original
Köhler, J., Klein, L., and Noé, F · 2019
Later among the works it cites.
Asymptotically unbiased estimation of physical observables with neural samplers
Original
Nicoli, K. A., Nakajima, S., Strodthoff, N., jciech Samek, W., Müller, K.-R., and Kessel, P · 2019
Later among the works it cites.
Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning
Noé, F., Olsson, S., Köhler, J., and Wu, H · 2019
Later among the works it cites.
Normalizing flows for probabilistic modeling and inference
Original
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B · 2019
Later among the works it cites.
Equivariant hamiltonian flows
Original
Rezende, D. J., Racanière, S., Higgins, I., and Toth, P · 2019
Later among the works it cites.
Hamiltonian generative networks
Original
Toth, P., Rezende, D. J., Jaegle, A., Racanière, S., Botev, A., and Higgins, I · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…