2020

Non-saturating GAN training as divergence minimization

Shannon, Matt, Poole, Ben, Mariooryad, Soroosh et al.

Understand

Non-saturating generative adversarial network (GAN) training is widely used and has continued to obtain groundbreaking results.

However so far this approach has lacked strong theoretical justification, in contrast to alternatives such as f-GANs and Wasserstein GANs which are motivated in terms of approximate divergence minimization.
In this paper we show that non-saturating GAN training does in fact approximately minimize a particular f-divergence.
We develop general theoretical tools to compare and classify f-divergences and use these to show that the new f-divergence is qualitatively similar to reverse KL.

Built on

Training GANs with centripetal acceleration
Original
Peng, W., Dai, Y., Zhang, H., and Cheng, L. (2019) · 1902
Earlier work this paper cites.
A general class of coefficients of divergence of one distribution from another
Ali, S. M. and Silvey, S. D. (1966) · 1966
Earlier work this paper cites.
Information-type measures of difference of probability distributions and indirect observation
Csiszár, I. (1967) · 1967
Earlier work this paper cites.
On the f-divergence and singularity of probability measures
Vajda, I. (1972) · 1972
Earlier work this paper cites.
Pattern recognition and machine learning
Bishop, C. M. (2006) · 2006
Earlier work this paper cites.
Comprehensive survey on distance/similarity measures between probability density functions
Cha, S.-H. (2007) · 2007
Earlier work this paper cites.
Properties of f-divergences and f-GAN training
Original
Shannon, M. (2020) · 2009
Earlier work this paper cites.
On metric divergences of probability measures
Vajda, I. (2009) · 2009
Earlier work this paper cites.
Information, divergence and risk for binary experiments
Reid, M. D. and Williamson, R. C. (2011) · 2011
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014) · 2014
Earlier work this paper cites.

Similar

Auto-encoding variational Bayes
Kingma, D. P. and Welling, M. (2014) · 2014
Cited alongside, same era.
f-GAN: Training generative neural samplers using variational divergence minimization
Nowozin, S., Cseke, B., and Tomioka, R. (2016) · 2016
Cited alongside, same era.
Improved generator objectives for GANs
Poole, B., Alemi, A. A., Sohl-Dickstein, J., and Angelova, A. (2016) · 2016
Cited alongside, same era.
f-divergence inequalities
Sason, I. and Verdú, S. (2016) · 2016
Cited alongside, same era.
Towards principled methods for training generative adversarial networks
Arjovsky, M. and Bottou, L. (2017) · 2017
Cited alongside, same era.
Wasserstein generative adversarial networks
Arjovsky, M., Chintala, S., and Bottou, L. (2017) · 2017
Cited alongside, same era.
Variance regularizing adversarial learning
Original
Grewal, K., Hjelm, R. D., and Bengio, Y. (2017) · 2017
Cited alongside, same era.
Improved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. (2017) · 2017
Cited alongside, same era.
On convergence and stability of GANs
Original
Kodali, N., Abernethy, J., Hays, J., and Kira, Z. (2017) · 2017
Cited alongside, same era.
The numerics of GANs
Mescheder, L., Nowozin, S., and Geiger, A. (2017) · 2017
Cited alongside, same era.

Then

Gradient descent GAN optimization is locally stable
Nagarajan, V. and Kolter, J. Z. (2017) · 2017
Later among the works it cites.
Amortised MAP inference for image super-resolution
Sønderby, C. K., Caballero, J., Theis, L., Shi, W., and Huszár, F. (2017) · 2017
Later among the works it cites.
The mechanics of n-player differentiable games
Balduzzi, D., Racaniere, S., Martens, J., Foerster, J., Tuyls, K., and Graepel, T. (2018) · 2018
Later among the works it cites.
Large scale GAN training for high fidelity natural image synthesis
Brock, A., Donahue, J., and Simonyan, K. (2018) · 2018
Later among the works it cites.
Many paths to equilibrium: GANs do not need to decrease a divergence at every step
Fedus, W., Rosca, M., Lakshminarayanan, B., Dai, A. M., Mohamed, S., and Goodfellow, I. (2018) · 2018
Later among the works it cites.
Progressive growing of GANs for improved quality, stability, and variation
Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2018) · 2018
Later among the works it cites.
Are GANs created equal? A large-scale study
Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bousquet, O. (2018) · 2018
Later among the works it cites.
Which training methods for GANs do actually converge?
Mescheder, L., Geiger, A., and Nowozin, S. (2018) · 2018
Later among the works it cites.
Spectral normalization for generative adversarial networks
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. (2018) · 2018
Later among the works it cites.
A style-based generator architecture for generative adversarial networks
Karras, T., Laine, S., and Aila, T. (2019) · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…