2020

Self-supervised Representation Learning for Evolutionary Neural Architecture Search

Wei, Chen, Tang, Yiping, Niu, Chuang et al.

Understand

Recently proposed neural architecture search (NAS) algorithms adopt neural predictors to accelerate the architecture search.

The capability of neural predictors to accurately predict the performance metrics of neural architecture is critical to NAS, and the acquisition of training datasets for neural predictors is time-consuming.
How to obtain a neural predictor with high prediction accuracy using a small amount of training data is a central problem to neural predictor-based NAS.
Here, we firstly design a new architecture encoding scheme that overcomes the drawbacks of existing vector-based architecture encoding schemes to calculate the graph edit distance of neural architectures.

Built on

C. White, W. Neiswanger, and Y. Savani. (2019) Bananas: Bayesian optimization with neural architectures for neural architecture search. [Online]. Available: https://arxiv.org/abs/1910.11858
Original
1910
Earlier work this paper cites.
Y. Bai, H. Ding, Y. Qiao, A. Marinovic, K. Gu, T. Chen, Y. Sun, and W. Wang, “Unsupervised inductive graph-level representation learning via graph-graph proximity,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019 , S. Kraus, Ed. ijcai.org, 2019, pp. 1988–1994
1994
Earlier work this paper cites.
T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. (2020) A simple framework for contrastive learning of visual representations. [Online]. Available: https://arxiv.org/abs/2002.05709
Original
2002
Earlier work this paper cites.
C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, and J. min Liang. (2020) Npenas: Neural predictor guided evolution for neural architecture search. [Online]. Available: https://arxiv.org/abs/2003.12857
Original
2003
Earlier work this paper cites.
S. Yan, Y. Zheng, W. Ao, X. Zeng, and M. Zhang. (2020) Does unsupervised architecture representation learning help neural architecture search? [Online]. Available: https://arxiv.org/abs/2006.06936
Original
2006
Earlier work this paper cites.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. (2020) Unsupervised learning of visual features by contrasting cluster assignments. [Online]. Available: https://arxiv.org/abs/2006.09882
Original
2006
Earlier work this paper cites.
H. Cheng, T. Zhang, S. Li, F. Yan, M. Li, V. Chandra, H. H. Li, and Y. Chen. (2020) NASGEM: neural architecture search via graph embedding method. [Online]. Available: https://arxiv.org/abs/2007.04452
Original
2007
Earlier work this paper cites.
A. Krizhevsky and G. E. Hinton, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009
2009
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , 2012, pp. 1106–1114
2012
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , Y. Bengio and Y. LeCun, Eds., 2014
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR , vol. abs/1412.6980, 2014
Original
2014
Earlier work this paper cites.
M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI , ser. Lecture Notes in Computer Science, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds., vol. 9910. Springer, 2016, pp. 69–84
2016
Earlier work this paper cites.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings , 2017
2017
Earlier work this paper cites.
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , 2017, pp. 8697–8710
2017
Earlier work this paper cites.
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , vol. 70, 2017, pp. 2902–2911
2017
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings , 2017
2017
Earlier work this paper cites.

Similar

T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” Journal of Machine Learning research , vol. 20, pp. 55:1–55:21, 2018
2018
Cited alongside, same era.
C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I , vol. 11205, 2018, pp. 19–35
2018
Cited alongside, same era.
S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018
2018
Cited alongside, same era.
H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , vol. 80, 2018, pp. 4092–4101
2018
Cited alongside, same era.
B. Baker, O. Gupta, R. Raskar, and N. Naik, “Accelerating neural architecture search using performance prediction,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings , 2018
2018
Cited alongside, same era.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , J. Burstein, C. Doran, and T. Solorio, Eds., 2019, pp. 4171–4186
2019
Cited alongside, same era.
C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “NAS-bench-101: Towards reproducible neural architecture search,” in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , vol. 97, 2019, pp. 7105–7114
2019
Cited alongside, same era.
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 , 2019, pp. 4780–4789
2019
Cited alongside, same era.
H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019
2019
Cited alongside, same era.
H. Zhou, M. Yang, J. Wang, and W. Pan, “Bayesnas: A bayesian approach for neural architecture search,” in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , vol. 97, 2019, pp. 7603–7613
2019
Cited alongside, same era.
S. Xie, H. Zheng, C. Liu, and L. Lin, “SNAS: stochastic neural architecture search,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019
2019
Cited alongside, same era.
X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap between search and evaluation,” pp. 1294–1303, 2019
2019
Cited alongside, same era.

Then

M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric,” in International Conference on Learning Representations Workshop on Representation Learning on Graphs and Manifolds , 2019
2019
Later among the works it cites.
L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” in Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019 , 2019, p. 129
2019
Later among the works it cites.
L. Wang, Y. Zhao, Y. Jinnai, Y. Tian, and R. Fonseca, “Neural architecture search using deep neural networks and monte carlo tree search,” in The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , 2020, pp. 9983–9991
2020
Closest in time.
X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang, “A generic graph-based neural architecture encoding scheme for predictor-based NAS,” in Proceedings of the European Conference on Computer Vision , 2020
2020
Closest in time.
W. Wen, H. Liu, Y. Chen, H. H. Li, G. Bender, and P. Kindermans, “Neural predictor for neural architecture search,” in Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX , A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12374. Springer, 2020, pp. 660–676
2020
Closest in time.
K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, “Momentum contrast for unsupervised visual representation learning,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 . IEEE, 2020, pp. 9726–9735
2020
Closest in time.
Y. M. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simultaneous clustering and representation learning,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020
2020
Closest in time.
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems , 2020
2020
Closest in time.
J. You, J. Leskovec, K. He, and S. Xie, “Graph structure of neural networks,” in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 12-18 July 2020 , 2020
2020
Closest in time.
X. Dong and Y. Yang, “Nas-bench-201: Extending the scope of reproducible neural architecture search,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020
2020
Closest in time.
Y. Sun, B. Xue, M. Zhang, G. G. Yen, and J. Lv, “Automatically designing cnn architectures using the genetic algorithm for image classification,” IEEE Transactions on Cybernetics , pp. 1–15, 2020
2020
Closest in time.
Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “Completely automated cnn architecture design based on blocks,” IEEE Transactions on Neural Networks and Learning Systems , vol. 31, no. 4, pp. 1242–1254, 2020
2020
Closest in time.
——, “Evolving deep convolutional neural networks for image classification,” IEEE Transactions on Evolutionary Computation , vol. 24, no. 2, pp. 394–407, 2020
2020
Closest in time.
Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, “PC-DARTS: Partial channel connections for memory-efficient architecture search,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020
2020
Closest in time.
C. Wei. (2020) Code for Self-supervised Represetnation Learning for Evolutionary Neural Architecture Search. [Online]. Available: https://github.com/auroua/SSNENAS
2020
Closest in time.
K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada , 2018, pp. 2016–2025
2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…