2022

Flow Matching for Generative Modeling

Lipman, Yaron, Chen, Ricky T. Q., Ben-Hamu, Heli et al.

Understand

We introduce a new paradigm for generative modeling built on Continuous Normalizing Flows (CNFs), allowing us to train CNFs at unprecedented scale.

Specifically, we present the notion of Flow Matching (FM), a simulation-free approach for training CNFs based on regressing vector fields of fixed conditional probability paths.
Flow Matching is compatible with a general family of Gaussian probability paths for transforming between noise and data samples -- which subsumes existing diffusion paths as specific instances.
Interestingly, we find that employing FM with diffusion paths results in a more robust and stable alternative for training diffusion models.

Built on

Potential flow generator with $l_2$ optimal transport regularity for generative models
Original
Liu Yang and George E. Karniadakis · 1908
Earlier work this paper cites.
A family of embedded runge-kutta formulae
John R Dormand and Peter J Prince · 1980
Earlier work this paper cites.
A convexity principle for interacting gases
Robert J McCann · 1997
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Optimal transport: old and new , volume 338
Cédric Villani · 2009
Earlier work this paper cites.
Denoising diffusion implicit models
Original
Jiaming Song, Chenlin Meng, and Stefano Ermon · 2010
Earlier work this paper cites.
Score-based generative modeling through stochastic differential equations
Original
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole · 2011
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
Pascal Vincent · 2011
Earlier work this paper cites.
Numerical solution of SDE through computer experiments
Peter Eris Kloeden, Eckhard Platen, and Henri Schurz · 2012
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
A downsampled variant of imagenet as an alternative to the cifar datasets
Original
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter · 2017
Earlier work this paper cites.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Earlier work this paper cites.
Ai and compute
Dario Amodei, Danny Hernandez, Girish SastryJack, Jack Clark, Greg Brockman, and Ilya Sutskever · 2018
Earlier work this paper cites.
torchdiffeq, 2018
Ricky T. Q. Chen · 2018
Earlier work this paper cites.
Neural ordinary differential equations
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018
Earlier work this paper cites.
Ffjord: Free-form continuous dynamics for scalable reversible generative models, 2018
Original
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud · 2018
Earlier work this paper cites.

Similar

Mgan: Training generative adversarial nets with multiple generators
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung · 2018
Cited alongside, same era.
Pacgan: The power of two samples in generative adversarial networks
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh · 2018
Cited alongside, same era.
Logo synthesis and manipulation with clustered generative adversarial networks
Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool · 2018
Cited alongside, same era.
Augmented neural odes
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh · 2019
Cited alongside, same era.
High-fidelity image generation with fewer labels
Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly · 2019
Cited alongside, same era.
Generative modeling by estimating gradients of the data distribution
Yang Song and Stefano Ermon · 2019
Cited alongside, same era.
How to train your neural ode: the world of jacobian and kinetic regularization
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M. Oberman · 2020
Cited alongside, same era.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Cited alongside, same era.
Learning differential equations that are easy to solve
Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud · 2020
Cited alongside, same era.
Normalizing flows: An introduction and review of current methods
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker · 2020
Cited alongside, same era.
The computational limits of deep learning
Original
Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso · 2020
Cited alongside, same era.
Trajectorynet: A dynamic optimal transport network for modeling cellular dynamics
Alexander Tong, Jessie Huang, Guy Wolf, David Van Dijk, and Smita Krishnaswamy · 2020
Cited alongside, same era.

Then

Improved denoising diffusion probabilistic models
Alexander Quinn Nichol and Prafulla Dhariwal · 2021
Later among the works it cites.
Ot-flow: Fast and accurate continuous normalizing flows via optimal transport
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto · 2021
Later among the works it cites.
Normalizing flows for probabilistic modeling and inference
George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan · 2021
Later among the works it cites.
Non-denoising forward-time diffusions
Stefano Peluchetti · 2021
Later among the works it cites.
Moser flow: Divergence-based generative modeling on manifolds
Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman · 2021
Later among the works it cites.
Maximum likelihood training of score-based diffusion models
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon · 2021
Later among the works it cites.
Deep generative learning via schrödinger bridge
Original
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang · 2021
Later among the works it cites.
Building normalizing flows with stochastic interpolants
Original
Michael S Albergo and Eric Vanden-Eijnden · 2022
Closest in time.
Matching normalizing flows and probability paths on manifolds
Original
Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Aditya Grover, Maximilian Nickel, Ricky T. Q. Chen, and Yaron Lipman · 2022
Closest in time.
To-flow: Efficient continuous normalizing flows with temporal optimization adjoint with moving speed, 2022
Original
Shian Du, Yihong Luo, Wei Chen, Jian Xu, and Delu Zeng · 2022
Closest in time.
Flow straight and fast: Learning to generate and transfer data with rectified flow
Original
Xingchao Liu, Chengyue Gong, and Qiang Liu · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents
Original
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Closest in time.
Image super-resolution via iterative refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi · 2022
Closest in time.
Fast sampling of diffusion models with exponential integrator
Original
Qinsheng Zhang and Yongxin Chen · 2022
Closest in time.
Action matching: A variational method for learning stochastic dynamics from samples, 2023
Kirill Neklyudov, Daniel Severo, and Alireza Makhzani · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…