2020

OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport

Onken, Derek, Fung, Samy Wu, Li, Xingjian et al.

Understand

A normalizing flow is an invertible mapping between an arbitrary probability distribution and a standard normal distribution; it can be used for density estimation and statistical inference.

Computing the flow follows the change of variables formula and thus requires invertibility of the mapping and an efficient way to compute the determinant of its Jacobian.
To satisfy these requirements, normalizing flows typically consist of carefully chosen components.
Continuous normalizing flows (CNFs) are mappings obtained by solving a neural ordinary differential equation (ODE).

Built on

Fluid Flow Mass Transport for Generative Networks
Original
Lin, J.; Lensink, K.; and Haber, E. 2019 · 1910
Earlier work this paper cites.
Normalizing Flows for Probabilistic Modeling and Inference
Original
Papamakarios, G.; Nalisnick, E.; Rezende, D. J.; Mohamed, S.; and Lakshminarayanan, B. 2019 · 1912
Earlier work this paper cites.
A Stochastic Estimator of the Trace of the Influence Matrix for Laplacian Smoothing Splines
Hutchinson, M. F. 1990 · 1990
Earlier work this paper cites.
The Geometry of Optimal Transportation
Gangbo, W.; and McCann, R. J. 1996 · 1996
Earlier work this paper cites.
Partial Differential Equations and Monge-Kantorovich Mass Transfer
Evans, L. C. 1997 · 1997
Earlier work this paper cites.
On-Line Learning Fokker-Planck Machine
Suykens, J.; Verrelst, H.; and Vandewalle, J. 1998 · 1998
Earlier work this paper cites.
A Computational Fluid Mechanics Solution to the Monge-Kantorovich Mass Transfer Problem
Benamou, J.-D.; and Brenier, Y. 2000 · 2000
Earlier work this paper cites.
Analysis of the Streamline Upwind/Petrov Galerkin Method Applied to the Solution of Optimal Control Problems
Collis, S. S.; and Heinkenschloss, M. 2002 · 2002
Earlier work this paper cites.
APAC-Net: Alternating the Population and Agent Control via Two Neural Networks to Solve High-Dimensional Stochastic Mean Field Games
Original
Lin, A. T.; Fung, S. W.; Li, W.; Nurbekyan, L.; and Osher, S. J. 2020 · 2002
Earlier work this paper cites.
The Effect of Stabilization in Finite Element Methods for the Optimal Boundary Control of the Oseen Equations
Abraham, F.; Behr, M.; and Heinkenschloss, M. 2004 · 2004
Earlier work this paper cites.
Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows
Original
Onken, D.; and Ruthotto, L. 2020 · 2005
Earlier work this paper cites.
Optimal Control of the Convection-Diffusion Equation using Stabilized Finite Element Methods
Becker, R.; and Vexler, B. 2007 · 2007
Earlier work this paper cites.
Gradient Flows: In Metric Spaces and in the Space of Probability Measures
Ambrosio, L.; Gigli, N.; and Savaré, G. 2008 · 2008
Earlier work this paper cites.
Numerical Methods for Evolutionary Differential Equations , volume 5
Ascher, U. M. 2008 · 2008
Earlier work this paper cites.
Optimal Transport: Old and New , volume 338
Villani, C. 2008 · 2008
Earlier work this paper cites.
Partial Differential Equations , volume 19
Evans, L. C. 2010 · 2010
Earlier work this paper cites.
Randomized Algorithms for Estimating the Trace of an Implicit Symmetric Positive Semi-Definite Matrix
Avron, H.; and Toledo, S. 2011 · 2011
Earlier work this paper cites.
MCMC using Hamiltonian Dynamics
Neal, R. M. 2011 · 2011
Earlier work this paper cites.
Bayesian Learning via Stochastic Gradient Langevin Dynamics
Welling, M.; and Teh, Y. W. 2011 · 2011
Earlier work this paper cites.

Similar

A Kernel Two-Sample Test
Gretton, A.; Borgwardt, K. M.; Rasch, M. J.; Schölkopf, B.; and Smola, A. 2012 · 2012
Cited alongside, same era.
An Introduction to Mathematical Optimal Control Theory Version 0.2
Evans, L. C. 2013 · 2013
Cited alongside, same era.
A Family of Nonparametric Density Estimation Algorithms
Tabak, E. G.; and Turner, C. V. 2013 · 2013
Cited alongside, same era.
Trends in PDE Constrained Optimization , volume 165
Leugering, G.; Benner, P.; Engell, S.; Griewank, A.; Harbrecht, H.; Hinze, M.; Rannacher, R.; and Ulbrich, S. 2014 · 2014
Cited alongside, same era.
NICE: Non-linear Independent Components Estimation
Dinh, L.; Krueger, D.; and Bengio, Y. 2015 · 2015
Cited alongside, same era.
MADE: Masked Autoencoder for Distribution Estimation
Germain, M.; Gregor, K.; Murray, I.; and Larochelle, H. 2015 · 2015
Cited alongside, same era.
Generative Moment Matching Networks
Li, Y.; Swersky, K.; and Zemel, R. 2015 · 2015
Cited alongside, same era.
Variational Inference with Normalizing Flows
Rezende, D. J.; and Mohamed, S. 2015 · 2015
Cited alongside, same era.
Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
Salimans, T.; Kingma, D.; and Welling, M. 2015 · 2015
Cited alongside, same era.
Deep Residual Learning for Image Recognition
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016 · 2016
Cited alongside, same era.
Improved Variational Inference with Inverse Autoregressive Flow
Kingma, D. P.; Salimans, T.; Jozefowicz, R.; Chen, X.; Sutskever, I.; and Welling, M. 2016 · 2016
Cited alongside, same era.
A Note on the Evaluation of Generative Models
Theis, L.; van den Oord, A.; and Bethge, M. 2016 · 2016
Cited alongside, same era.

Then

On the Convergence and Robustness of Training GANs with Regularized Optimal Transport
Sanjabi, M.; Ba, J.; Razaviyayn, M.; and Lee, J. D. 2018 · 2018
Later among the works it cites.
Monge-Ampère Flow for Generative Modeling
Original
Zhang, L.; E, W.; and Wang, L. 2018 · 2018
Later among the works it cites.
Parallel Optimal Transport GAN
Avraham, G.; Zuo, Y.; and Drummond, T. 2019 · 2019
Later among the works it cites.
Neural Spline Flows
Durkan, C.; Bekasov, A.; Murray, I.; and Papamakarios, G. 2019 · 2019
Later among the works it cites.
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
Gholaminejad, A.; Keutzer, K.; and Biros, G. 2019 · 2019
Later among the works it cites.
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
Grathwohl, W.; Chen, R. T.; Betterncourt, J.; Sutskever, I.; and Duvenaud, D. 2019 · 2019
Later among the works it cites.
Learning Protein Structure with a Differentiable Simulator
Ingraham, J.; Riesselman, A.; Sander, C.; and Marks, D. 2019 · 2019
Later among the works it cites.
A Geometric View of Optimal Transportation and Generative Model
Lei, N.; Su, K.; Cui, L.; Yau, S.-T.; and Gu, X. D. 2019 · 2019
Later among the works it cites.
Boltzmann Generators: Sampling Equilibrium States of Many-Body Systems with Deep Learning
Noé, F.; Olsson, S.; Köhler, J.; and Wu, H. 2019 · 2019
Later among the works it cites.
Computational Optimal Transport
Peyré, G.; and Cuturi, M. 2019 · 2019
Later among the works it cites.
Discriminator Optimal Transport
Tanaka, A. 2019 · 2019
Later among the works it cites.
Unconstrained Monotonic Neural Networks
Wehenkel, A.; and Louppe, G. 2019 · 2019
Later among the works it cites.
MadMiner: Machine Learning-Based Inference for Particle Physics
Brehmer, J.; Kling, F.; Espejo, I.; and Cranmer, K. 2020 · 2020
Closest in time.
How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization
Finlay, C.; Jacobsen, J.-H.; Nurbekyan, L.; and Oberman, A. M. 2020 · 2020
Closest in time.
Normalizing Flows: An Introduction and Review of Current Methods
Kobyzev, I.; Prince, S.; and Brubaker, M. 2020 · 2020
Closest in time.
A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems
Ruthotto, L.; Osher, S. J.; Li, W.; Nurbekyan, L.; and Fung, S. W. 2020 · 2020
Closest in time.
Potential Flow Generator With L 2 {L}_{2} Optimal Transport Regularity for Generative Models
Yang, L.; and Karniadakis, G. E. 2020 · 2020
Closest in time.
Neural Autoregressive Flows
Huang, C.-W.; Krueger, D.; Lacoste, A.; and Courville, A. 2018 · 2087
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…