2020

Riemannian Continuous Normalizing Flows

Mathieu, Emile, Nickel, Maximilian

Understand

Normalizing flows have shown great promise for modelling flexible probability distributions in a computationally tractable way.

However, whilst data is often naturally described on Riemannian manifolds such as spheres, torii, and hyperbolic spaces, most normalizing flows implicitly assume a flat geometry, making them either misspecified or ill-suited in these situations.
To overcome this problem, we introduce Riemannian continuous normalizing flows, a model which admits the parametrization of flexible probability measures on smooth manifolds by defining flows as the solution to ordinary differential equations.
We show that this approach can lead to substantial improvements on both synthetic and real-world data when compared to standard flows or previously introduced projected flows.

Built on

Normalizing Flows for Probabilistic Modeling and Inference
Original
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2019) · 1912
Earlier work this paper cites.
A derivation of n-Dimensional spherical coordinates
Blumenson, L. E. (1960) · 1960
Earlier work this paper cites.
Orientational statistics
Downs (1972) · 1972
Earlier work this paper cites.
A family of embedded Runge-Kutta formulae
Dormand, R. J. and Prince, J. P. (1980) · 1980
Earlier work this paper cites.
A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines
Hutchinson (1990) · 1990
Earlier work this paper cites.
Improving Generalization Performance Using Double Backpropagation
Drucker, H. and Cun, Y. L. (1992) · 1992
Earlier work this paper cites.
Harmonic analysis and the global exponential map for compact Lie groups
Dooley, A. and Wildberger, N. (1993) · 1993
Earlier work this paper cites.
A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem
Benamou, J.-D. and Brenier, Y. (2000) · 2000
Earlier work this paper cites.
Directional Statistics
Mardia, K. V. and Jupp, P. E. (2000) · 2000
Earlier work this paper cites.
Optimal transport maps in Monge-Kantorovich problem
Ambrosio, L. (2003) · 2003
Earlier work this paper cites.
Introduction to Smooth Manifolds
Lee, J. M. (2003) · 2003
Earlier work this paper cites.
Neural Ordinary Differential Equations on Manifolds
Original
Falorsi, L. and Forré, P. (2020) · 2006
Earlier work this paper cites.
Geometric Numerical Integration : Structure-Preserving Algorithms for Ordinary Differential Equations
Hairer, E. (2006) · 2006
Earlier work this paper cites.
Sampling Realistic Protein Conformations Using Local Structural Bias
Hamelryck, T., Kent, J. T., and Krogh, A. (2006) · 2006
Earlier work this paper cites.
Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements
Pennec, X. (2006) · 2006
Earlier work this paper cites.
Riemannian Geometry
Petersen, P. (2006) · 2006
Earlier work this paper cites.
A gyrovector space approach to hyperbolic geometry
Ungar, A. A. (2008) · 2008
Earlier work this paper cites.
A New Scaling and Squaring Algorithm for the Matrix Exponential
Al-Mohy, A. H. and Higham, N. J. (2010) · 2010
Earlier work this paper cites.

Similar

Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix
Avron, H. and Toledo, S. (2011) · 2011
Cited alongside, same era.
Solving Differential Equations on Manifolds
Hairer, E. (2011) · 2011
Cited alongside, same era.
A Jacobian Inequality for Gradient Maps on the Sphere and Its Application to Directional Statistics
Sei, T. (2011) · 2011
Cited alongside, same era.
Advances in matrix manifolds for computer vision
Lui, Y. M. (2012) · 2012
Cited alongside, same era.
Stochastic gradient descent on Riemannian manifolds
Bonnabel, S. (2013) · 2013
Cited alongside, same era.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M. (2014) · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014) · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J. (2015) · 2015
Cited alongside, same era.
Variational inference with normalizing flows
Rezende, D. and Mohamed, S. (2015) · 2015
Cited alongside, same era.
Normalizing Flows on Riemannian Manifolds
Original
Gemici, M. C., Rezende, D., and Mohamed, S. (2016) · 2016
Cited alongside, same era.
Global active archive of large flood events
Brakenridge, G. (2017) · 2017
Cited alongside, same era.
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017) · 2017
Cited alongside, same era.

Then

Hyperbolic neural networks
Ganea, O., Becigneul, G., and Hofmann, T. (2018) · 2018
Later among the works it cites.
Sensitivity and generalization in neural networks: an empirical study
Novak, R., Bahri, Y., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J. (2018) · 2018
Later among the works it cites.
Deep Diffeomorphic Normalizing Flows
Original
Salman, H., Yadollahpour, P., Fletcher, T., and Batmanghelich, K. (2018) · 2018
Later among the works it cites.
Riemannian adaptive optimization methods
Becigneul, G. and Ganea, O.-E. (2019) · 2019
Later among the works it cites.
Residual flows for invertible generative modeling
Chen, R. T. Q., Behrmann, J., Duvenaud, D. K., and Jacobsen, J.-H. (2019) · 2019
Later among the works it cites.
Reparameterizing distributions on lie groups
Falorsi, L., de Haan, P., Davidson, T. R., and Forré, P. (2019) · 2019
Later among the works it cites.
Scalable reversible generative models with free-form continuous dynamics
Grathwohl, W., Chen, R. T. Q., Bettencourt, J., and Duvenaud, D. (2019) · 2019
Later among the works it cites.
Machine learning for the geosciences: Challenges and opportunities
Karpatne, A., Ebert-Uphoff, I., Ravela, S., Babaie, H. A., and Kumar, V. (2019) · 2019
Later among the works it cites.
Continuous hierarchical representations with poincaré variational auto-encoders
Mathieu, E., Le Lan, C., Maddison, C. J., Tomioka, R., and Teh, Y. W. (2019) · 2019
Later among the works it cites.
A Differentiable Gaussian-like Distribution on Hyperbolic Space for Gradient-Based Learning
Nagano, Y., Yamaguchi, S., Fujita, Y., and Koyama, M. (2019) · 2019
Later among the works it cites.
Latent variable modelling with hyperbolic normalizing flows
Bose, A. J., Smofsky, A., Liao, R., Panangaden, P., and Hamilton, W. L. (2020) · 2020
Closest in time.
Active fire data
EOSDIS (2020) · 2020
Closest in time.
How to train your neural ODE
Finlay, C., Jacobsen, J.-H., Nurbekyan, L., and Oberman, A. M. (2020) · 2020
Closest in time.
Poincaré maps for analyzing complex hierarchies in single-cell data
Klimovskaia, A., Lopez-Paz, D., Bottou, L., and Nickel, M. (2020) · 2020
Closest in time.
Neural manifold ordinary differential equations
Lou, A., Lim, D., Katsman, I., Huang, L., Jiang, Q., Lim, S.-N., and De Sa, C. (2020) · 2020
Closest in time.
Normalizing Flows on Tori and Spheres
Rezende, D. J., Papamakarios, G., Racanière, S., Albergo, M. S., Kanwar, G., Shanahan, P. E., and Cranmer, K. (2020) · 2020
Closest in time.
Mixed-curvature variational autoencoders
Skopek, O., Ganea, O.-E., and Bécigneul, G. (2020) · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…