2022

Riemannian Score-Based Generative Modelling

De Bortoli, Valentin, Mathieu, Emile, Hutchinson, Michael et al.

Understand

Score-based generative models (SGMs) are a powerful class of generative models that exhibit remarkable empirical performance.

Score-based generative modelling (SGM) consists of a ``noising'' stage, whereby a diffusion is used to gradually add Gaussian noise to data, and a generative model, which entails a ``denoising'' process defined by approximating the time-reversal of the diffusion.
Existing SGMs assume that data is supported on a Euclidean space, i.e.
a manifold with flat geometry.

Built on

“From the Schrödinger problem to the Monge–Kantorovich problem”
Christian Léonard · 1920
Earlier work this paper cites.
“Sur la théorie relativiste de l’électron et l’interprétation de la mécanique quantique”
Erwin Schrödinger · 1932
Earlier work this paper cites.
“A Special Stokes’s Theorem for Complete Riemannian Manifolds”
Matthew. Gaffney · 1954
Earlier work this paper cites.
“The central limit problem for geodesic random walks”
Erik Jørgensen · 1975
Earlier work this paper cites.
“Convergence of time-inhomogeneous geodesic random walks and its application to coupling methods”
Kazumasa Kuwada · 1979
Earlier work this paper cites.
“Some isoperimetric inequalities and eigenvalue estimates”
Christopher Croke · 1980
Earlier work this paper cites.
“A Family of Embedded Runge-Kutta Formulae”
R.. Dormand and J.. Prince · 1980
Earlier work this paper cites.
“Large deviations and the Malliavin calculus”
Jean-Michel Bismut · 1984
Earlier work this paper cites.
“Eigenvalues in Riemannian Geometry”
Isaac Chavel · 1984
Earlier work this paper cites.
“Time reversal of diffusions”
Ulrich Haussmann and Etienne Pardoux · 1986
Earlier work this paper cites.
“Large time behavior of the heat equation on complete manifolds with non-negative Ricci curvature”
Peter Li · 1986
Earlier work this paper cites.
“A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines”
Michael Hutchinson · 1989
Earlier work this paper cites.
“Stochastic Differential Equations and Diffusion Processes” 24
Nobuyuki Ikeda and Shinzo Watanabe · 1989
Earlier work this paper cites.
“Isometric embeddings of Riemannian manifolds, Kyoto, 1990”
Matthias Gunther · 1991
Earlier work this paper cites.
“Logarithmic Sobolev inequalities on Lie groups”
Leonard Gross · 1992
Earlier work this paper cites.
“Precise estimates on the rate at which certain diffusions tend to equilibrium”
Laurent Saloff-Coste · 1994
Earlier work this paper cites.
“Stratonovich stochastic differential equations driven by general semimartingales”
Thomas Kurtz, Étienne Pardoux and Philip Protter · 1995
Earlier work this paper cites.
“Information Theory and Statistics” Reprint of the second (1968) edition
Solomon Kullback · 1997
Earlier work this paper cites.
“Estimates of heat kernels on Riemannian manifolds”
Alexander Grigor’yan · 1999
Earlier work this paper cites.
“Estimates of derivatives of the heat kernel on a compact Riemannian manifold”
Elton Hsu · 1999
Earlier work this paper cites.
“Continuous Martingales and Brownian Motion” 293
Daniel Revuz and Marc Yor · 1999
Earlier work this paper cites.
“Statistics of Random Processes. I” 5
Robert. Liptser and Albert. Shiryaev · 2001
Earlier work this paper cites.
“Fitting mixtures of Kent distributions to aid in joint set identification”
David Peel, William Whiten and Geoffrey McLachlan · 2001
Earlier work this paper cites.
“Stochastic Analysis on Manifolds”
Elton Hsu · 2002
Earlier work this paper cites.
“Optimal Transport Maps in Monge-Kantorovich Problem”
Luigi Ambrosio · 2003
Earlier work this paper cites.
“Estimation of non-normalized statistical models by score matching.”
Aapo Hyvärinen · 2005
Earlier work this paper cites.
“The large-scale structure of semantic networks: Statistical analyses and a model of semantic growth”
Mark Steyvers and Joshua Tenenbaum · 2005
Earlier work this paper cites.
“Riemannian Manifolds: An Introduction to Curvature”
John Lee · 2006
Earlier work this paper cites.
“Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements”
Xavier Pennec · 2006
Earlier work this paper cites.
“Convergence rates to equilibrium of the heat kernels on compact Riemannian manifolds”
Hajime Urakawa · 2006
Earlier work this paper cites.
“Learning annotated hierarchies from relational data”
Daniel Roy, Charles Kemp, Vikash Mansinghka and Joshua B · 2007
Earlier work this paper cites.
“Manifold parametrizations by eigenfunctions of the Laplacian and Heat Kernels”
Peter. Jones, Mauro Maggioni and Raanan Schul · 2008
Earlier work this paper cites.
“Introduction to Topological Manifolds”
John Lee · 2010
Earlier work this paper cites.
“Pose Estimation Using Linearized Rotations and Quaternion Algebra”
Timothy Barfoot, James. Forbes and Paul. Furgale · 2011
Earlier work this paper cites.
“Numerical Solution of Stochastic Differential Equations”, Stochastic Modelling and Applied Probability
P.E. Kloeden and E. Platen · 2011
Earlier work this paper cites.
“A smoothed backbone-dependent rotamer library for proteins derived from adaptive kernel density estimates and regressions”
Maxim Shapovalov and Roland Dunbrack · 2011
Earlier work this paper cites.

Similar

“A connection between score matching and denoising autoencoders”
Pascal Vincent · 2011
Cited alongside, same era.
“Numerical Continuation Methods: An Introduction”
Eugene Allgower and Kurt Georg · 2012
Cited alongside, same era.
“Spherical Harmonics and Approximations on the Unit Sphere: An Introduction”
Kendall Atkinson and Weimin Han · 2012
Cited alongside, same era.
“A Kernel Two-Sample Test”
Arthur Gretton, Karsten. Borgwardt, Malte. Rasch, Bernhard Schölkopf and Alexander Smola · 2012
Cited alongside, same era.
“Girsanov theory under a finite entropy condition”
Christian Léonard · 2012
Cited alongside, same era.
“Advances in matrix manifolds for computer vision”
Yui Lui · 2012
Cited alongside, same era.
“Rigid motion estimation using mixtures of projected Gaussians”
Wendelin Feiten, Muriel Lang and Sandra Hirche · 2013
Cited alongside, same era.
“A lower bound for the first eigenvalue in the Laplacian operator on compact Riemannian manifolds”
Yue He · 2013
Cited alongside, same era.
“Smooth Manifolds”
John Lee · 2013
Cited alongside, same era.
“A Jacobian inequality for gradient maps on the sphere and its application to directional statistics”
Tomonari Sei · 2013
Cited alongside, same era.
“Analysis and Geometry of Markov Diffusion Operators”
D. Bakry, I. Gentil and M. Ledoux · 2014
Cited alongside, same era.
“Geometric Measure Theory”
Herbert Federer · 2014
Cited alongside, same era.

Then

“Denoising diffusion probabilistic models”
Jonathan Ho, Ajay Jain and Pieter Abbeel · 2020
Later among the works it cites.
“Poincaré maps for analyzing complex hierarchies in single-cell data”
Anna Klimovskaia, David Lopez-Paz, Léon Bottou and Maximilian Nickel · 2020
Later among the works it cites.
“Equivariant Flows: Exact Likelihood Generative Learning for Symmetric Densities”
Original
Jonas Köhler, Leon Klein and Frank Noé · 2020
Later among the works it cites.
“Riemannian Continuous Normalizing Flows”
Emile Mathieu and Maximilian Nickel · 2020
Later among the works it cites.
“Geomstats: A Python Package for Riemannian Geometry in Machine Learning”
Nina Miolane, Nicolas Guigui, Alice Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes and Xavier Pennec · 2020
Later among the works it cites.
“Improved techniques for training score-based generative models”
Yang Song and Stefano Ermon · 2020
Later among the works it cites.
“Conditional Image Generation with Score-Based Diffusion Models”
Original
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb and Christian Etmann · 2021
Later among the works it cites.
“Rectangular flows for manifold learning”
Original
Anthony. Caterini, Gabriel Loaiza-Ganem, Geoff Pleiss and John. Cunningham · 2021
Later among the works it cites.
“Time reversal of diffusion processes under a finite entropy condition”
Original
Patrick Cattiaux, Giovanni Conforti, Ivan Gentil and Christian Léonard · 2021
Later among the works it cites.
“Logarithmic heat kernels: estimates without curvature restrictions”
Original
Xin Chen, Xue Li and Bo Wu · 2021
Later among the works it cites.
“Density Ratio Estimation via Infinitesimal Classification”
Original
Kristy Choi, Chenlin Meng, Yang Song and Stefano Ermon · 2021
Later among the works it cites.
“Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction”
Original
Hyungjin Chung, Byeongsu Sim and Jong Ye · 2021
Later among the works it cites.
“Riemannian Convex Potential Maps”
Original
Samuel Cohen, Brandon Amos and Yaron Lipman · 2021
Later among the works it cites.
“Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling”
Valentin De, James Thornton, Jeremy Heng and Arnaud Doucet · 2021
Later among the works it cites.
“Diffusion models beat GAN on Image Synthesis”
Original
Prafulla Dhariwal and Alex Nichol · 2021
Later among the works it cites.
“Continuous Normalizing Flows on Manifolds”
Original
Luca Falorsi · 2021
Later among the works it cites.
“Brenier–Schródinger problem on compact manifolds with boundary”
David García-Zelada and Baptiste Huguet · 2021
Later among the works it cites.
“Adversarial score matching and improved sampling for image generation”
Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet and Ioannis Mitliagkas · 2021
Later among the works it cites.
“SNIPS: Solving Noisy Inverse Problems Stochastically”
Original
Bahjat Kawar, Gregory Vaksman and Michael Elad · 2021
Later among the works it cites.
“Stochastic Image Denoising by Sampling from the Posterior Distribution”
Original
Bahjat Kawar, Gregory Vaksman and Michael Elad · 2021
Later among the works it cites.
“PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior”
Original
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon and Tie-Yan Liu · 2021
Later among the works it cites.
“Existence, uniqueness and regularity of the projection onto differentiable manifolds”
Gunther Leobacher and Alexander Steinicke · 2021
Later among the works it cites.
“Improved denoising diffusion probabilistic models”
Original
Alex Nichol and Prafulla Dhariwal · 2021
Later among the works it cites.
“Implicit Riemannian concave potential maps”
Original
Danilo. Rezende and Sébastien Racanière · 2021
Later among the works it cites.
“Moser Flow: Divergence-based Generative Modeling on Manifolds”
Noam Rozen, Aditya Grover, Maximilian Nickel and Yaron Lipman · 2021
Later among the works it cites.
“D2C: Diffusion-Denoising Models for Few-shot Conditional Generation”
Original
Abhishek Sinha, Jiaming Song, Chenlin Meng and Stefano Ermon · 2021
Later among the works it cites.
“Score-Based Generative Modeling through Stochastic Differential Equations”
Yang Song, Jascha Sohl-Dickstein, Diederik. Kingma, Abhishek Kumar, Stefano Ermon and Ben Poole · 2021
Later among the works it cites.
“Solving Schrödinger Bridges via Maximum Likelihood”
Original
Francisco Vargas, Pierre Thodoroff, Neil. Lawrence and Austen Lamacraft · 2021
Later among the works it cites.
“Learning to Efficiently Sample from Diffusion Probabilistic Models”
Original
Daniel Watson, Jonathan Ho, Mohammad Norouzi and William Chan · 2021
Later among the works it cites.
“NCEI/WDS Global Significant Earthquake Database”, https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ngdc.mgg.hazards:G012153, 2022
National/ (NGDC/WDS) · 2022
Closest in time.
“NCEI/WDS Global Significant Volcanic Eruptions Database”, https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ngdc.mgg.hazards:G10147, 2022
National/ (NGDC/WDS) · 2022
Closest in time.
“Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models”
Original
Fan Bao, Chongxuan Li, Jun Zhu and Bo Zhang · 2022
Closest in time.
“Theory and Algorithms for Diffusion Processes on Riemannian Manifolds”
Original
Xiang Cheng, Jingzhao Zhang and Suvrit Sra · 2022
Closest in time.
“Independent SE(3)-Equivariant Models for End-to-End Rigid Protein Docking”
Octavian-Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi. Jaakkola and Andreas Krause · 2022
Closest in time.
“Riemannian Diffusion Schrodinger Bridge”
Original
James Thornton, Michael Hutchinson, Emile Mathieu, Valentin De, Yee Teh and Arnaud Doucet · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…