2018

Generative Adversarial Network Training is a Continual Learning Problem

Liang, Kevin J, Li, Chunyuan, Wang, Guoyin et al.

Understand

Generative Adversarial Networks (GANs) have proven to be a powerful framework for learning to draw samples from complex distributions.

However, GANs are also notoriously difficult to train, with mode collapse and oscillations a common problem.
We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator.
Recognizing this, our contributions are twofold.

Built on

Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
Michael McCloskey and Neal J Cohen · 1989
Earlier work this paper cites.
Connectionist Models of Recognition Memory: Constraints Imposed by Learning and Forgetting Functions
Roger Ratcliff · 1990
Earlier work this paper cites.
BLEU: a Method for Automatic Evaluation of Machine Translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio · 2013
Earlier work this paper cites.
Generative Adversarial Networks
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Conditional Generative Adversarial Nets
Mehdi Mirza and Simon Osindero · 2014
Earlier work this paper cites.
Microsoft COCO Captions: Data Collection and Evaluation Server
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dolí, and C Lawrence Zitnick · 2015
Earlier work this paper cites.
Distilling the Knowledge in a Neural Network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
An Empirical Evaluation of Deep Learning on Highway Driving
Brody Huval, Tao Wang, Sameep Tandon, Jeff Kiske, Will Song, Joel Pazhayampallil, Mykhaylo Andriluka, Pranav Rajpurkar, Toki Migimatsu, Royce Cheng-Yue, Fernando Mujica, Adam Coates, and Andrew Y. Ng · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis · 2015
Earlier work this paper cites.
Categorical reparameterization with gumbel-softmax
Original
Eric Jang, Shixiang Gu, and Ben Poole · 2016
Earlier work this paper cites.
Autoencoding beyond pixels using a learned similarity metric
Anders Larsen, Søren Sønderby, Hugo Larochelle, and Ole Winther · 2016
Earlier work this paper cites.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Alec Radford, Luke Metz, and Soumith Chintala · 2016
Earlier work this paper cites.
Improved Techniques for Training GANs
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen · 2016
Earlier work this paper cites.

Similar

Mode regularized generative adversarial networks
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li · 2017
Cited alongside, same era.
Generative Multi-Adversarial Networks
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan · 2017
Cited alongside, same era.
Improved Training of Wasserstein GANs
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville · 2017
Cited alongside, same era.
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Cited alongside, same era.
Overcoming Catastrophic Forgetting in Neural Networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell · 2017
Cited alongside, same era.
Adversarial Learning for Neural Dialogue Generation
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky · 2017
Cited alongside, same era.
Adversarial ranking for language generation
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Ting Sun · 2017
Cited alongside, same era.
The Numerics of GANs
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger · 2017
Cited alongside, same era.
Unrolled Generative Adversarial Networks
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein · 2017
Cited alongside, same era.
Gradient descent GAN optimization is locally stable
Vaishnavh Nagarajan and J Zico Kolter · 2017
Cited alongside, same era.
Stabilizing GAN Training with Multiple Random Projections
Behnam Neyshabur, Srinadh Bhojanapalli, and Ayan Chakrabarti · 2017
Cited alongside, same era.
Bayesian GAN
Yunus Saatchi and Andrew Gordan Wilson · 2017
Cited alongside, same era.

Then

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu · 2017
Later among the works it cites.
Continual Learning Through Synaptic Intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli · 2017
Later among the works it cites.
Adversarial feature matching for text generation
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin · 2017
Later among the works it cites.
Towards Robust Evaluations of Continual Learning
Sebastian Farquhar and Yarin Gal · 2018
Closest in time.
Multi-Agent Diverse Generative Adversarial Networks
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Iit Kanpur, Philip H S Torr, and Puneet K Dokania · 2018
Closest in time.
Long Text Generation via Adversarial Training with Leaked Information
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang · 2018
Closest in time.
MGAN: Training Generative Adversarial Nets with Multiple Generators
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung · 2018
Closest in time.
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2018
Closest in time.
Automatic Threat Recognition of Prohibited Items at Aviation Checkpoints with X-Ray Imaging: a Deep Learning Approach
Kevin J Liang, Geert Heilmann, Christopher Gregory, Souleymane Diallo, David Carlson, Gregory Spell, John Sigman, Kris Roe, and Lawrence Carin · 2018
Closest in time.
Spectral Normalization for Generative Adversarial Networks
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida · 2018
Closest in time.
Continual Learning in Generative Adversarial Nets
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu · 2018
Closest in time.
On catastrophic forgetting and mode collapse in Generative Adversarial Networks
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh · 2018
Closest in time.
Coulomb GANs: Provably Optimal Nash Equilibria Via Potential Fields
Thomas Unterthiner, Bernhard Nessler, Calvin Seward, Günter Klambauer, Martin Heusel, Hubert Ramsauer, and Sepp Hochreiter · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…