2022

TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed

Du, Shian, Luo, Yihong, Chen, Wei et al.

Understand

Continuous normalizing flows (CNFs) construct invertible mappings between an arbitrary complex distribution and an isotropic Gaussian distribution using Neural Ordinary Differential Equations (neural ODEs).

It has not been tractable on large datasets due to the incremental complexity of the neural ODE training.
Optimal Transport theory has been applied to regularize the dynamics of the ODE to speed up training in recent works.
In this paper, a temporal optimization is proposed by optimizing the evolutionary time for forward propagation of the neural ODE training.

Built on

A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines
Michael F Hutchinson · 1989
Earlier work this paper cites.
Mnist handwritten digit database, 1998
Yann LeCun, Corinna Cortes, and C Burges · 1998
Earlier work this paper cites.
A computational fluid mechanics solution to the monge-kantorovich mass transfer problem
Jean-David Benamou and Yann Brenier · 2000
Earlier work this paper cites.
Frobnication
FirstName Alpher · 2002
Earlier work this paper cites.
Frobnication revisited
FirstName Alpher and FirstName Fotheringham-Smythe · 2003
Earlier work this paper cites.
Can a machine frobnicate?
FirstName Alpher, FirstName Fotheringham-Smythe, and FirstName Gamow · 2004
Earlier work this paper cites.
Can a computer frobnicate?
FirstName Alpher and FirstName Gamow · 2005
Earlier work this paper cites.
Learning multiple layers of features from tiny images.(2009), 2009
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
A family of nonparametric density estimation algorithms
Esteban G Tabak and Cristina V Turner · 2013
Earlier work this paper cites.
Real and Convex Analysis
Erhan Çınlar and Robert J Vanderbei · 2013
Earlier work this paper cites.
The frobnicatable foo filter, 2014
FirstName LastName · 2014
Earlier work this paper cites.
Frobnication tutorial, 2014
FirstName LastName · 2014
Earlier work this paper cites.
Stochastic gradient vb and the variational auto-encoder
Diederik P Kingma and Max Welling · 2014
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Nice: Non-linear independent components estimation
Original
Laurent Dinh, David Krueger, and Yoshua Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Deep residual learning
Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al · 2015
Earlier work this paper cites.
Task loss estimation for sequence prediction
Original
Dzmitry Bahdanau, Dmitriy Serdyuk, Philemon Brakel, Nan Rosemary Ke, Jan Chorowski, Aaron Courville, and Yoshua Bengio · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Danilo Rezende and Shakir Mohamed · 2015
Earlier work this paper cites.
Markov chain monte carlo and variational inference: Bridging the gap
Tim Salimans, Diederik Kingma, and Max Welling · 2015
Earlier work this paper cites.
Density estimation using real nvp
Original
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio · 2016
Earlier work this paper cites.
Improved variational inference with inverse autoregressive flow
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Masked autoregressive flow for density estimation
Original
George Papamakarios, Theo Pavlakou, and Iain Murray · 2017
Earlier work this paper cites.
Maximum principle based algorithms for deep learning
Original
Qianxiao Li, Long Chen, Cheng Tai, et al · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Original
Han Xiao, Kashif Rasul, and Roland Vollgraf · 2017
Earlier work this paper cites.

Similar

Glow: Generative flow with invertible 1x1 convolutions
Original
Diederik P Kingma and Prafulla Dhariwal · 2018
Cited alongside, same era.
Flowavenet: A generative flow for raw audio
Original
Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon · 2018
Cited alongside, same era.
Neural ordinary differential equations
Original
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud · 2018
Cited alongside, same era.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud · 2018
Cited alongside, same era.
Neural autoregressive flows
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville · 2018
Cited alongside, same era.
Flow++: Improving flow-based generative models with variational dequantization and architecture design
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel · 2019
Cited alongside, same era.
Noise flow: Noise modeling with conditional normalizing flows
Abdelrahman Abdelhamed, Marcus A Brubaker, and Michael S Brown · 2019
Cited alongside, same era.
Videoflow: A flow-based generative model for video
Original
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma · 2019
Cited alongside, same era.
Universal audio synthesizer control with normalizing flows
Original
Philippe Esling, Naotake Masuda, Adrien Bardet, Romeo Despres, et al · 2019
Cited alongside, same era.
Waveglow: A flow-based generative network for speech synthesis
Ryan Prenger, Rafael Valle, and Bryan Catanzaro · 2019
Cited alongside, same era.
Graphnvp: An invertible flow model for generating molecular graphs
Original
Kaushalya Madhawa, Katushiko Ishiguro, Kosuke Nakago, and Motoki Abe · 2019
Cited alongside, same era.
Improving exploration in soft-actor-critic with normalizing flows policies
Original
Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose · 2019
Cited alongside, same era.

Then

Bsp-net: Generating compact meshes via binary space partitioning
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang · 2020
Later among the works it cites.
Leveraging exploration in off-policy algorithms via normalizing flows
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm · 2020
Later among the works it cites.
Randomized value functions via multiplicative normalizing flows
Ahmed Touati, Harsh Satija, Joshua Romoff, Joelle Pineau, and Pascal Vincent · 2020
Later among the works it cites.
Normalizing flows on tori and spheres
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racanière, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer · 2020
Later among the works it cites.
Equivariant flows: exact likelihood generative learning for symmetric densities
Jonas Köhler, Leon Klein, and Frank Noé · 2020
Later among the works it cites.
Targeted free energy estimation via learned mappings
Peter Wirnsberger, Andrew J Ballard, George Papamakarios, Stuart Abercrombie, Sébastien Racanière, Alexander Pritzel, Danilo Jimenez Rezende, and Charles Blundell · 2020
Later among the works it cites.
Gravitational-wave population inference with deep flow-based generative network
Kaze WK Wong, Gabriella Contardo, and Shirley Ho · 2020
Later among the works it cites.
Adaptive checkpoint adjoint method for gradient estimation in neural ode
Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan · 2020
Later among the works it cites.
Ot-flow: Fast and accurate continuous normalizing flows via optimal transport
Original
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto · 2020
Later among the works it cites.
Normalizing flows: An introduction and review of current methods
Ivan Kobyzev, Simon Prince, and Marcus Brubaker · 2020
Later among the works it cites.
Time dependence in non-autonomous neural odes
Original
Jared Quincy Davis, Krzysztof Choromanski, Jake Varley, Honglak Lee, Jean-Jacques Slotine, Valerii Likhosterov, Adrian Weller, Ameesh Makadia, and Vikas Sindhwani · 2020
Later among the works it cites.
How to train your neural ode: the world of jacobian and kinetic regularization
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman · 2020
Later among the works it cites.
Accelerating continuous normalizing flow with trajectory polynomial regularization
Original
Han-Hsien Huang and Mi-Yen Yeh · 2020
Later among the works it cites.
Steer: Simple temporal regularization for neural odes
Original
Arnab Ghosh, Harkirat Singh Behl, Emilien Dupont, Philip HS Torr, and Vinay Namboodiri · 2020
Later among the works it cites.
Vid-ode: Continuous-time video generation with neural ordinary differential equation
Original
Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jaegul Choo, Joonseok Lee, Sookyung Kim, and Edward Choi · 2020
Later among the works it cites.
Model-based reinforcement learning for semi-markov decision processes with neural odes
Original
Jianzhun Du, Joseph Futoma, and Finale Doshi-Velez · 2020
Later among the works it cites.
Stochasticity in neural odes: An empirical study
Original
Viktor Oganesyan, Alexandra Volokhova, and Dmitry Vetrov · 2020
Later among the works it cites.
Riemannian continuous normalizing flows
Original
Emile Mathieu and Maximilian Nickel · 2020
Later among the works it cites.
Layer-parallel training of deep residual neural networks
Stefanie Gunther, Lars Ruthotto, Jacob B Schroder, Eric C Cyr, and Nicolas R Gauger · 2020
Later among the works it cites.
Learning differential equations that are easy to solve
Original
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud · 2020
Later among the works it cites.
E (n) equivariant normalizing flows for molecule generation in 3d
Original
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B Fuchs, Ingmar Posner, and Max Welling · 2021
Later among the works it cites.
Improving robustness and uncertainty modelling in neural ordinary differential equations
Srinivas Anumasa and PK Srijith · 2021
Later among the works it cites.
Mali: A memory efficient and reverse accurate integrator for neural odes
Original
Juntang Zhuang, Nicha C Dvornek, Sekhar Tatikonda, and James S Duncan · 2021
Later among the works it cites.
Multi-scale neural odes for 3d medical image registration
Original
Junshen Xu, Eric Z Chen, Xiao Chen, Terrence Chen, and Shanhui Sun · 2021
Later among the works it cites.
Gaussian processes meet neuralodes: A bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data
Original
Mohamed Aziz Bhouri and Paris Perdikaris · 2021
Later among the works it cites.
Coupled graph ode for learning interacting system dynamics
Zijie Huang, Yizhou Sun, and Wei Wang · 2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…