2018

MD-GAN: Multi-Discriminator Generative Adversarial Networks for Distributed Datasets

Hardy, Corentin, Merrer, Erwan Le, Sericola, Bruno

Understand

A recent technical breakthrough in the domain of machine learning is the discovery and the multiple applications of Generative Adversarial Networks (GANs).

Those generative models are computationally demanding, as a GAN is composed of two deep neural networks, and because it trains on large datasets.
A GAN is generally trained on a single server.
In this paper, we address the problem of distributing GANs so that they are able to train over datasets that are spread on multiple workers.

Built on

Y. LeCun, C. Cortes, and C. J. Burges, “The mnist database of handwritten digits,” http://yann.lecun.com/exdb/mnist , 1998. [Online]. Available: http://yann.lecun.com/exdb/mnist/
1998
Earlier work this paper cites.
M. J. Weinberger, G. Seroussi, and G. Sapiro, “The loco-i lossless image compression algorithm: principles and standardization into jpeg-ls,” IEEE Transactions on Image Processing , vol. 9, no. 8, pp. 1309–1324, Aug 2000
2000
Earlier work this paper cites.
G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels, “Dynamo: Amazon’s highly available key-value store,” in SOSP , 2007
2007
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009
2009
Earlier work this paper cites.
B. Recht, C. Re, S. Wright, and F. Niu, “Hogwild: A lock-free approach to parallelizing stochastic gradient descent,” in NIPS , 2011
2011
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng, “Large scale distributed deep networks,” in NIPS , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012
2012
Earlier work this paper cites.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Networks,” ArXiv e-prints , Jun. 2014
2014
Earlier work this paper cites.
M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, “Scaling distributed machine learning with the parameter server,” in OSDI , 2014
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR , vol. abs/1412.6980, 2014
Original
2014
Earlier work this paper cites.
T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, “Project adam: Building an efficient and scalable deep learning training system,” in OSDI , 2014
2014
Earlier work this paper cites.
X. Lian, Y. Huang, Y. Li, and J. Liu, “Asynchronous parallel stochastic gradient for nonconvex optimization,” in NIPS , C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates, Inc., 2015. [Online]. Available: http://papers.nips.cc/paper/5751-asynchronous-parallel-stochastic-gradient-for-nonconvex-optimization.pdf
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in CVPR , 2015
2015
Earlier work this paper cites.
Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in Proceedings of International Conference on Computer Vision (ICCV) , 2015
2015
Earlier work this paper cites.
S. Zhang, A. E. Choromanska, and Y. LeCun, “Deep learning with elastic averaging SGD,” in NIPS , C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds., 2015
2015
Earlier work this paper cites.
R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in CCS , 2015
2015
Earlier work this paper cites.
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative Adversarial Text to Image Synthesis,” ArXiv e-prints , May 2016
2016
Earlier work this paper cites.

Similar

C. Vondrick, H. Pirsiavash, and A. Torralba, “Generating Videos with Scene Dynamics,” ArXiv e-prints , Sep. 2016
2016
Cited alongside, same era.
G. Perarnau, J. van de Weijer, B. Raducanu, and J. M. Álvarez, “Invertible Conditional GANs for image editing,” ArXiv e-prints , Nov. 2016
2016
Cited alongside, same era.
I. Cano, M. Weimer, D. Mahajan, C. Curino, and G. M. Fumarola, “Towards geo-distributed machine learning,” CoRR , vol. abs/1603.09035, 2016
Original
2016
Cited alongside, same era.
I. Durugkar, I. Gemp, and S. Mahadevan, “Generative Multi-Adversarial Networks,” 5th International Conference on Learning Representations (ICLR 2017) , Nov. 2016
2016
Cited alongside, same era.
W. Zhang, S. Gupta, X. Lian, and J. Liu, “Staleness-aware async-sgd for distributed deep learning,” 2016
2016
Cited alongside, same era.
H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” CoRR , vol. abs/1602.05629, 2016
Original
2016
Cited alongside, same era.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved Techniques for Training GANs,” ArXiv e-prints , Jun. 2016
2016
Cited alongside, same era.
M. Blot, D. Picard, M. Cord, and N. Thome, “Gossip training for deep learning,” ArXiv e-prints , Nov. 2016
2016
Cited alongside, same era.
Y. Wang, L. Zhang, and J. van de Weijer, “Ensembles of generative adversarial networks,” in NIPS 2016 Workshop on Adversarial Training , 2016
2016
Cited alongside, same era.
J. Konečný, H. Brendan McMahan, F. X. Yu, P. Richtárik, A. Theertha Suresh, and D. Bacon, “Federated Learning: Strategies for Improving Communication Efficiency,” CoRR , vol. abs/1610.05492, Oct. 2016
Original
2016
Cited alongside, same era.
J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting Distributed Synchronous SGD,” arXiv e-prints , p. arXiv:1604.00981, Apr. 2016
Original
2016
Cited alongside, same era.
S. Gupta, W. Zhang, and F. Wang, “Model accuracy and runtime tradeoff in distributed deep learning: A systematic study,” in ICDM , Dec 2016
2016
Cited alongside, same era.

Then

C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network,” CVPR , 2017
2017
Later among the works it cites.
M. Chidambaram and Y. Qi, “Style transfer generative adversarial networks: Learning to play chess differently,” CoRR , vol. abs/1702.06762, 2017
Original
2017
Later among the works it cites.
K. Hsieh, A. Harlap, N. Vijaykumar, D. Konomis, G. R. Ganger, P. B. Gibbons, and O. Mutlu, “Gaia: Geo-distributed machine learning approaching LAN speeds,” in NSDI , 2017
2017
Later among the works it cites.
Q. Hoang, T. Dinh Nguyen, T. Le, and D. Phung, “Multi-Generator Generative Adversarial Nets,” ArXiv e-prints , Aug. 2017
2017
Later among the works it cites.
C. Hardy, E. Le Merrer, and B. Sericola, “Distributed deep learning on edge-devices: Feasibility via adaptive compression,” in NCA , 2017
2017
Later among the works it cites.
M. Arjovsky and L. Bottou, “Towards Principled Methods for Training Generative Adversarial Networks,” ArXiv e-prints , Jan. 2017
2017
Later among the works it cites.
M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” ArXiv e-prints , Jan. 2017
2017
Later among the works it cites.
A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier GANs,” in Proceedings of the 34th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. International Convention Centre, Sydney, Australia: PMLR, 06–11 Aug 2017, pp. 2642–2651
2017
Later among the works it cites.
I. O. Tolstikhin, S. Gelly, O. Bousquet, C.-J. SIMON-GABRIEL, and B. Schölkopf, “Adagan: Boosting generative models,” in Advances in Neural Information Processing Systems 30 , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5424–5433. [Online]. Available: http://papers.nips.cc/paper/7126-adagan-boosting-generative-models.pdf
2017
Later among the works it cites.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” ArXiv e-prints , Jun. 2017
2017
Later among the works it cites.
K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy preserving machine learning,” Cryptology ePrint Archive, Report 2017/281, 2017
2017
Later among the works it cites.
I. Tolstikhin, S. Gelly, O. Bousquet, C.-J. Simon-Gabriel, and B. Schölkopf, “AdaGAN: Boosting Generative Models,” arXiv e-prints , p. arXiv:1701.02386, Jan. 2017
Original
2017
Later among the works it cites.
P. Blanchard, E. M. E. Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in NIPS , 2017
2017
Later among the works it cites.
E. J. Hyunsun Choi, “Generative ensembles for robust anomaly detection,” CoRR , vol. abs/1810.01392v1, 2018
Original
2018
Closest in time.
C. Hardy, E. Le Merrer, and B. Sericola, “Gossiping GANs,” in Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning , ser. DIDL ’18, 2018
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…