2020

On Leveraging Pretrained GANs for Generation with Limited Data

Zhao, Miaoyun, Cong, Yulai, Carin, Lawrence

Understand

Recent work has shown generative adversarial networks (GANs) can generate highly realistic images, that are often indistinguishable (by humans) from real images.

Most images so generated are not contained in the training dataset, suggesting potential for augmenting training sets with GAN-generated data.
While this scenario is of particular relevance when there are limited data available, there is still the issue of training the GAN itself based on that limited data.
To facilitate this, we leverage existing GAN models pretrained on large-scale datasets (like ImageNet) to introduce additional knowledge (which may not exist within the limited data), following the concept of transfer learning.

Built on

Analyzing and improving the image quality of stylegan
Original
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T · 1912
Earlier work this paper cites.
Learning many related tasks at the same time with backpropagation
Caruana, R · 1995
Earlier work this paper cites.
Automated flower classification over a large number of classes
Nilsback, M. and Zisserman, A · 2008
Earlier work this paper cites.
Deep learning of representations for unsupervised and transfer learning
Bengio, Y · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
3D object representations for fine-grained categorization
Krause, J., Stark, M., J., D., and Fei-Fei, L · 2013
Earlier work this paper cites.
Overfeat: Integrated recognition, localization and detection using convolutional networks
Original
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y · 2013
Earlier work this paper cites.
Decaf: A deep convolutional activation feature for generic visual recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Girshick, R., Donahue, J., Darrell, T., and Malik, J · 2014
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Zeiler, M. and Fergus, R · 2014
Earlier work this paper cites.
Learning deep features for scene recognition using places database
Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A · 2014
Earlier work this paper cites.
Deep learning face attributes in the wild
Liu, Z., Luo, P., Wang, X., and Tang, X · 2015
Earlier work this paper cites.
Learning transferable features with deep adaptation networks
Original
Long, M., Cao, Y., Wang, J., and Jordan, M · 2015
Earlier work this paper cites.
Deep multi-scale video prediction beyond mean square error
Original
Mathieu, M., Couprie, C., and LeCun, Y · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Network dissection: Quantifying interpretability of deep visual representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A · 2017
Earlier work this paper cites.
GANs trained by a two time-scale update rule converge to a local Nash equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S · 2017
Earlier work this paper cites.
Cycada: Cycle-consistent adversarial domain adaptation
Original
Hoffman, J., Tzeng, E., Park, T., Zhu, J., Isola, P., Saenko, K., Efros, A., and Darrell, T · 2017
Earlier work this paper cites.

Similar

Arbitrary style transfer in real-time with adaptive instance normalization
Huang, X. and Belongie, S · 2017
Cited alongside, same era.
Progressive growing of gans for improved quality, stability, and variation
Original
Karras, T., Aila, T., Laine, S., and Lehtinen, J · 2017
Cited alongside, same era.
Photo-realistic single image super-resolution using a generative adversarial network
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al · 2017
Cited alongside, same era.
Adversarial ranking for language generation
Lin, K., Li, D., He, X., Zhang, Z., and Sun, M · 2017
Cited alongside, same era.
Label efficient learning of transferable representations acrosss domains and tasks
Luo, Z., Zou, Y., Hoffman, J., and Fei-Fei, L · 2017
Cited alongside, same era.
The effectiveness of data augmentation in image classification using deep learning
Wang, J. and Perez, L · 2017
Cited alongside, same era.
A-fast-RCNN: Hard positive generation via adversary for object detection
Wang, X., Shrivastava, A., and Gupta, A · 2017
Cited alongside, same era.
GAN dissection: Visualizing and understanding generative adversarial networks
Original
Bau, D., Zhu, J., Strobelt, H., Zhou, B., Tenenbaum, J., Freeman, W., and Torralba, A · 2018
Cited alongside, same era.
Learning implicit brain MRI manifolds with deep learning
Bermudez, C., Plassard, A., Davis, L., Newton, A., Resnick, S., and Landman, B · 2018
Cited alongside, same era.
GAN augmentation: augmenting training data using generative adversarial networks
Original
Bowles, C., Chen, L., Guerrero, R., Bentley, P., Gunn, R., Hammers, A., Dickie, D., Hernández, M., Wardlaw, J., and Rueckert, D · 2018
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
MaskGAN: better text generation via filling in the _
Original
Fedus, W., Goodfellow, I., and Dai, A · 2018
Cited alongside, same era.

Then

Attribute manipulation generative adversarial networks for fashion images
Ak, K., Lim, J., Tham, J., and Kassim, A · 2019
Later among the works it cites.
Transfer learning from pre-trained bert for pronoun resolution
Bao, X. and Qiao, Q · 2019
Later among the works it cites.
Large scale GAN training for high fidelity natural image synthesis
Brock, A., Donahue, J., and Simonyan, K · 2019
Later among the works it cites.
Everybody dance now
Chan, C., Ginosar, S., Zhou, T., and Efros, A · 2019
Later among the works it cites.
GO gradient for expectation-based objectives
Cong, Y., Zhao, M., Bai, K., and Carin, L · 2019
Later among the works it cites.
Gansynth: Adversarial neural audio synthesis
Original
Engel, J., Agrawal, K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A · 2019
Later among the works it cites.
Mind2mind: transfer learning for GANs
Original
Frégier, Y. and Gouray, J · 2019
Later among the works it cites.
Transfer brain MRI tumor segmentation models across modalities with adversarial networks
Original
Giacomello, E., Loiacono, D., and Mainardi, L · 2019
Later among the works it cites.
Learning more with less: conditional PGGAN-based data augmentation for brain metastases detection using highly-rough annotation on MR images
Original
Han, C., Murao, K., Noguchi, T., Kawata, Y., Uchiyama, F., Rundo, L., Nakayama, H., and Satoh, S · 2019
Later among the works it cites.
MelGAN: Generative adversarial networks for conditional waveform synthesis
Original
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W., Sotelo, J., de Brebisson, A., Bengio, Y., and Courville, A · 2019
Later among the works it cites.
A BERT-based transfer learning approach for hate speech detection in online social media
Original
Mozafari, M., Farahbakhsh, R., and Crespi, N · 2019
Later among the works it cites.
Image generation from small datasets via batch statistics adaptation
Noguchi, A. and Harada, T · 2019
Later among the works it cites.
Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets
Original
Peng, Y., Yan, S., and Lu, Z · 2019
Later among the works it cites.
Open event extraction from online text using a generative adversarial network
Original
Wang, R., Zhou, D., and He, Y · 2019
Later among the works it cites.
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation
Original
Yamamoto, R., Song, E., and Kim, J · 2019
Later among the works it cites.
Generative adversarial network in medical imaging: A review
Yi, X., Walia, E., and Babyn, P · 2019
Later among the works it cites.
Infinite brain MR images: PGGAN-based data augmentation for tumor detection
Han, C., Rundo, L., Araki, R., Furukawa, Y., Mauri, G., Nakayama, H., and Hayashi, H · 2020
Closest in time.
Freeze discriminator: A simple baseline for fine-tuning GANs
Original
Mo, S., Cho, M., and Shin, J · 2020
Closest in time.
MineGAN: effective knowledge transfer from GANs to target domains with few images
Wang, Y., Gonzalez-Garcia, A., Berga, D., Herranz, L., Khan, F., and van de Weijer, J · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…