2021

Deep Generative Learning via Schr\"{o}dinger Bridge

Wang, Gefei, Jiao, Yuling, Xu, Qian et al.

Understand

We propose to learn a generative model via entropy interpolation with a Schr\"{o}dinger Bridge.

The generative learning task can be formulated as interpolating between a reference distribution and a target distribution based on the Kullback-Leibler divergence.
At the population level, this entropy interpolation is characterized via an SDE on $[0,1]$ with a time-varying drift term.
At the sample level, we derive our Schr\"{o}dinger Bridge algorithm by plugging the drift term estimated by a deep score estimator and a deep density ratio estimator into the Euler-Maruyama method.

Built on

Sur la théorie relativiste de l’électron et l’interprétation de la mécanique quantique
Schrödinger, E · 1932
Earlier work this paper cites.
The markov processes of schrödinger
Jamison, B · 1975
Earlier work this paper cites.
A stochastic control approach to reciprocal diffusion processes
Dai Pra, P · 1991
Earlier work this paper cites.
An algorithmic introduction to numerical simulation of stochastic differential equations
Higham, D. J · 2001
Earlier work this paper cites.
Rademacher and gaussian complexities: Risk bounds and structural results
Bartlett, P. L. and Mendelson, S · 2002
Earlier work this paper cites.
Neural network learning: Theoretical foundations
Anthony, M. and Bartlett, P. L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al · 2009
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
Vincent, P · 2011
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Kingma, D. P. and Welling, M · 2014
Earlier work this paper cites.
A survey of the schrodinger problem and some of its connections with optimal transport
Léonard, C · 2014
Earlier work this paper cites.
NICE: Non-linear independent components estimation
Dinh, L., Krueger, D., and Bengio, Y · 2015
Earlier work this paper cites.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks
Original
Radford, A., Metz, L., and Chintala, S · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Adversarial autoencoders
Makhzani, A., Shlens, J., Jaitly, N., and Goodfellow, I · 2016
Earlier work this paper cites.
f f -GAN: Training generative neural samplers using variational divergence minimization
Nowozin, S., Cseke, B., and Tomioka, R · 2016
Earlier work this paper cites.
Improved techniques for training gans
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X · 2016
Earlier work this paper cites.

Similar

WaveNet: A generative model for raw audio
Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. W., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.
Generative visual manipulation on the natural image manifold
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., and Efros, A. A · 2016
Cited alongside, same era.
Wasserstein generative adversarial networks
Arjovsky, M., Chintala, S., and Bottou, L · 2017
Cited alongside, same era.
Mode regularized generative adversarial networks
Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W · 2017
Cited alongside, same era.
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., and Bengio, S · 2017
Cited alongside, same era.
Improved training of Wasserstein gans
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C · 2017
Cited alongside, same era.
GANs trained by a two time-scale update rule converge to a local nash equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S · 2017
Cited alongside, same era.
Masked autoregressive flow for density estimation
Papamakarios, G., Pavlakou, T., and Murray, I · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A · 2017
Cited alongside, same era.
On gradient regularizers for MMD GANs
Arbel, M., Sutherland, D., Bińkowski, M., and Gretton, A · 2018
Cited alongside, same era.

Then

High-dimensional probability: An introduction with applications in data science , volume 47
Vershynin, R · 2018
Later among the works it cites.
Self-attention generative adversarial networks
Original
Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A · 2018
Later among the works it cites.
Maximum mean discrepancy gradient flow
Arbel, M., Korba, A., Salim, A., and Gretton, A · 2019
Later among the works it cites.
Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks
Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A · 2019
Later among the works it cites.
Invertible residual networks
Behrmann, J., Grathwohl, W., Chen, R. T. Q., Duvenaud, D., and Jacobsen, J.-H · 2019
Later among the works it cites.
Deep generative learning via variational gradient flow
Gao, Y., Jiao, Y., Wang, Y., Wang, Y., Yang, C., and Zhang, S · 2019
Later among the works it cites.
A style-based generator architecture for generative adversarial networks
Karras, T., Laine, S., and Aila, T · 2019
Later among the works it cites.
Sliced-Wasserstein flows: Nonparametric generative modeling via optimal transport and diffusions
Liutkus, A., Simsekli, U., Majewski, S., Durmus, A., Stöter, F.-R., Chaudhuri, K., and Salakhutdinov, R · 2019
Later among the works it cites.
Waveglow: A flow-based generative network for speech synthesis
Prenger, R., Valle, R., and Catanzaro, B · 2019
Later among the works it cites.
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., van den Oord, A., and Vinyals, O · 2019
Later among the works it cites.
Deep network approximation characterized by number of neurons
Original
Shen, Z., Yang, H., and Zhang, S · 2019
Later among the works it cites.
Generative modeling by estimating gradients of the data distribution
Song, Y. and Ermon, S · 2019
Later among the works it cites.
Generative modeling with denoising auto-encoders and langevin sampling
Original
Block, A., Mroueh, Y., and Rakhlin, A · 2020
Later among the works it cites.
Stochastic control liasons: Richard sinkhorn meets gaspard monge on a schroedinger bridge
Original
Chen, Y., Georgiou, T. T., and Pavon, M · 2020
Later among the works it cites.
Stargan v2: Diverse image synthesis for multiple domains
Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W · 2020
Later among the works it cites.
Generative learning with euler particle transport
Original
Gao, Y., Huang, J., Jiao, Y., Liu, J., Lu, X., and Yang, Z · 2020
Later among the works it cites.
Interpreting the latent space of gans for semantic face editing
Shen, Y., Gu, J., Tang, X., and Zhou, B · 2020
Later among the works it cites.
Improved techniques for training score-based generative models
Song, Y. and Ermon, S · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

Deep Generative Learning via Schr\&#34;{o}dinger Bridge

Understand

Built on

Similar

Then

Beyond the bibliography

Built on

Similar

Then

Beyond the bibliography

Deep Generative Learning via Schr\"{o}dinger Bridge