2021

Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)

Stanczuk, Jan, Etmann, Christian, Kreusser, Lisa Maria et al.

Understand

Wasserstein GANs are based on the idea of minimising the Wasserstein distance between a real and a generated distribution.

We provide an in-depth mathematical analysis of differences between the theoretical setup and the reality of training Wasserstein GANs.
In this work, we gather both theoretical and empirical evidence that the WGAN loss is not a meaningful approximation of the Wasserstein distance.
Moreover, we argue that the Wasserstein distance is not even a desirable loss function for deep generative models, and conclude that the success of Wasserstein GANs can in truth be attributed to a failure to approximate the Wasserstein distance.

Built on

(q,p)-wasserstein gans: Comparing ground metrics for wasserstein gans, 2019a
Original
Mallasto, A., Frellsen, J., Boomsma, W., and Feragen, A · 1902
Earlier work this paper cites.
Sur le point pour lequel la somme des distances de n points donnés est minimum
Weiszfeld, E · 1937
Earlier work this paper cites.
Least squares quantization in pcm
Lloyd, S · 1982
Earlier work this paper cites.
Distortion mismatch in the quantization of probability measures
Graf, S., Luschgy, H., and Pagès, G · 2008
Earlier work this paper cites.
From gan to wgan, 2019
Weng, L · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Sample complexity of testing the manifold hypothesis
Narayanan, H. and Mitter, S · 2010
Earlier work this paper cites.
Scikit-learn: Machine learning in python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al · 2011
Earlier work this paper cites.
Learning probability measures with respect to optimal transport metrics, 2012
Canas, G. D. and Rosasco, L · 2012
Earlier work this paper cites.
Sinkhorn distances: Lightspeed computation of optimal transportation distances, 2013
Cuturi, M · 2013
Earlier work this paper cites.
Generative adversarial networks, 2014
Original
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015
Earlier work this paper cites.
Unsupervised representation learning with deep convolutional generative adversarial networks, 2016
Radford, A., Metz, L., and Chintala, S · 2016
Earlier work this paper cites.
Towards principled methods for training generative adversarial networks, 2017
Original
Arjovsky, M. and Bottou, L · 2017
Earlier work this paper cites.
Wasserstein gan, 2017
Original
Arjovsky, M., Chintala, S., and Bottou, L · 2017
Earlier work this paper cites.
The cramer distance as a solution to biased wasserstein gradients, 2017
Bellemare, M. G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., and Munos, R · 2017
Earlier work this paper cites.

Similar

Began: Boundary equilibrium generative adversarial networks, 2017
Berthelot, D., Schumm, T., and Metz, L · 2017
Cited alongside, same era.
Inductive bias of deep convolutional networks through pooling geometry, 2017
Cohen, N. and Shashua, A · 2017
Cited alongside, same era.
Many paths to equilibrium: Gans do not need to decrease a divergence at every step, 2017
Original
Fedus, W., Rosca, M., Lakshminarayanan, B., Dai, A. M., Mohamed, S., and Goodfellow, I · 2017
Cited alongside, same era.
Learning generative models with sinkhorn divergences, 2017
Original
Genevay, A., Peyré, G., and Cuturi, M · 2017
Cited alongside, same era.
Improved training of wasserstein gans, 2017
Original
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A · 2017
Cited alongside, same era.
On convergence and stability of gans, 2017
Original
Kodali, N., Abernethy, J., Hays, J., and Kira, Z · 2017
Cited alongside, same era.
Are gans created equal? a large-scale study, 2017
Original
Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O · 2017
Cited alongside, same era.
Least squares generative adversarial networks, 2017
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., and Smolley, S. P · 2017
Cited alongside, same era.
The spatial inductive bias of deep learning, 2017
Mitchell, B · 2017
Cited alongside, same era.
Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance, 2017
Weed, J. and Bach, F · 2017
Cited alongside, same era.
Sorting out lipschitz function approximation, 2018
Original
Anil, C., Lucas, J., and Grosse, R · 2018
Cited alongside, same era.
Generative modeling using the sliced wasserstein distance, 2018
Original
Deshpande, I., Zhang, Z., and Schwing, A · 2018
Cited alongside, same era.

Then

A gradual, semi-discrete approach to generative network training via explicit wasserstein minimization, 2019
Chen, Y., Telgarsky, M., Zhang, C., Bailey, B., Hsu, D., and Peng, J · 2019
Later among the works it cites.
Wasserstein gan can perform pca
Cho, J. and Suh, C · 2019
Later among the works it cites.
Sample complexity of sinkhorn divergences, 2019
Genevay, A., Chizat, L., Bach, F., Cuturi, M., and Peyré, G · 2019
Later among the works it cites.
Manifold-valued image generation with wasserstein generative adversarial nets, 2019
Huang, Z., Wu, J., and Gool, L. V · 2019
Later among the works it cites.
Deep multi-wasserstein unsupervised domain adaptation
Le, T.-N., Habrard, A., and Sebban, M · 2019
Later among the works it cites.
Deep multi-wasserstein unsupervised domain adaptation
Le, T.-N., Habrard, A., and Sebban, M · 2019
Later among the works it cites.
Adversarial regularizers in inverse problems, 2019
Lunz, S., Öktem, O., and Schönlieb, C.-B · 2019
Later among the works it cites.
Subspace robust wasserstein distances, 2019
Paty, F.-P. and Cuturi, M · 2019
Later among the works it cites.
On the estimation of the wasserstein distance in generative models, 2019
Pinetz, T., Soukup, D., and Pock, T · 2019
Later among the works it cites.
Experiment tracking with weights and biases, 2020
Biewald, L · 2020
Later among the works it cites.
Learning with minibatch wasserstein : asymptotic and gradient properties, 2020
Fatras, K., Zine, Y., Flamary, R., Gribonval, R., and Courty, N · 2020
Later among the works it cites.
Computational optimal transport, 2020
Peyré, G. and Cuturi, M · 2020
Later among the works it cites.
Implicit competitive regularization in gans, 2020
Schäfer, F., Zheng, H., and Anandkumar, A · 2020
Later among the works it cites.
Deep image prior
Ulyanov, D., Vedaldi, A., and Lempitsky, V · 2020
Later among the works it cites.
Pot: Python optimal transport
Flamary, R., Courty, N., Gramfort, A., Alaya, M. Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N. T., Janati, H., Rakotomamonjy, A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, D. J., Tavenard, R., Tong, A., and Vayer, T · 2021
Closest in time.
The intrinsic dimension of images and its impact on learning, 2021
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…