2021

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

Li, Changlin, Tang, Tao, Wang, Guangrun et al.

Understand

A myriad of recent breakthroughs in hand-crafted neural architectures for visual recognition have highlighted the urgent need to explore hybrid architectures consisting of diversified building blocks.

Meanwhile, neural architecture search methods are surging with an expectation to reduce human efforts.
However, whether NAS methods can efficiently and effectively handle diversified search spaces with disparate candidates (e.g.
CNNs and transformers) is still an open question.

Built on

A new measure of rank correlation
Maurice G Kendall · 1938
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky and G. Hinton · 2009
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Unsupervised learning of visual representations by solving jigsaw puzzles
Mehdi Noroozi and Paolo Favaro · 2016
Earlier work this paper cites.
Convolutional neural fabrics
Shreyas Saxena and Jakob Verbeek · 2016
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Earlier work this paper cites.
Colorful image colorization
Richard Zhang, Phillip Isola, and Alexei A Efros · 2016
Earlier work this paper cites.
Designing neural network architectures using reinforcement learning
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar · 2017
Earlier work this paper cites.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam · 2017
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Deeparchitect: Automatically designing and training deep architectures
Original
Renato Negrinho and Geoffrey J. Gordon · 2017
Earlier work this paper cites.
Scaling sgd batch size to 32k for imagenet training
Original
Yang You, Igor Gitman, and Boris Ginsburg · 2017
Earlier work this paper cites.
Neural architecture search with reinforcement learning
Barret Zoph and Quoc V. Le · 2017
Earlier work this paper cites.
Understanding and simplifying one-shot architecture search
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc V. Le · 2018
Earlier work this paper cites.
SMASH: one-shot model architecture search through hypernetworks
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston · 2018
Earlier work this paper cites.
Squeeze-and-excitation networks
Jie Hu, Li Shen, and Gang Sun · 2018
Earlier work this paper cites.
Unsupervised representation learning by predicting image rotations
Nikos Komodakis and Spyros Gidaris · 2018
Earlier work this paper cites.
Progressive neural architecture search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy · 2018
Earlier work this paper cites.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Earlier work this paper cites.
Image transformer
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran · 2018
Earlier work this paper cites.
Efficient neural architecture search via parameters sharing
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean · 2018
Earlier work this paper cites.
Efficient neural architecture search via parameter sharing
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean · 2018
Earlier work this paper cites.
Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen · 2018
Earlier work this paper cites.
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He · 2018
Earlier work this paper cites.
Unsupervised feature learning via non-parametric instance discrimination
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin · 2018
Earlier work this paper cites.
Practical block-wise neural network architecture generation
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu · 2018
Earlier work this paper cites.
Learning transferable architectures for scalable image recognition
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le · 2018
Earlier work this paper cites.
Adaptive stochastic natural gradient method for one-shot neural architecture search
Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, and Kouhei Nishida · 2019
Earlier work this paper cites.
Proxylessnas: Direct neural architecture search on target task and hardware
Han Cai, Ligeng Zhu, and Song Han · 2019
Earlier work this paper cites.
RENAS: reinforced evolutionary neural architecture search
Yukang Chen, Gaofeng Meng, Qian Zhang, Shiming Xiang, Chang Huang, Lisen Mu, and Xinggang Wang · 2019
Earlier work this paper cites.

Similar

Searching for a robust neural architecture in four GPU hours
Xuanyi Dong and Yi Yang · 2019
Cited alongside, same era.
Learning deep representations by mutual information estimation and maximization
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio · 2019
Cited alongside, same era.
Searching for MobileNetV3
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam · 2019
Cited alongside, same era.
Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation
Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-Fei · 2019
Cited alongside, same era.
DARTS: differentiable architecture search
Hanxiao Liu, Karen Simonyan, and Yiming Yang · 2019
Cited alongside, same era.
Regularized evolution for image classifier architecture search
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le · 2019
Cited alongside, same era.
Meal: Multi-model ensemble via adversarial learning
Zhiqiang Shen, Zhankui He, and Xiangyang Xue · 2019
Cited alongside, same era.
Mnasnet: Platform-aware neural architecture search for mobile
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le · 2019
Cited alongside, same era.
Efficientnet: Rethinking model scaling for convolutional neural networks
Mingxing Tan and Quoc V. Le · 2019
Cited alongside, same era.
Contrastive multiview coding
Original
Yonglong Tian, Dilip Krishnan, and Phillip Isola · 2019
Cited alongside, same era.
Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer · 2019
Cited alongside, same era.
Nas-bench-101: Towards reproducible neural architecture search
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter · 2019
Cited alongside, same era.

Then

Evaluating the search phase of neural architecture search
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann · 2020
Later among the works it cites.
Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks
Original
Zhiqiang Shen and Marios Savvides · 2020
Later among the works it cites.
Bridging the gap between sample-based and one-shot neural architecture search with bonas
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang · 2020
Later among the works it cites.
Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al · 2020
Later among the works it cites.
Self-supervised representation learning for evolutionary neural architecture search
Original
Chen Wei, Yiping Tang, Chuang Niu, Haihong Hu, Yue Wang, and Jimin Liang · 2020
Later among the works it cites.
Does unsupervised architecture representation learning help neural architecture search?
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang · 2020
Later among the works it cites.
NAS evaluation is frustratingly hard
Antoine Yang, Pedro M. Esperança, and Fabio Maria Carlucci · 2020
Later among the works it cites.
Differentiable neural architecture search in equivalent space with exploration enhancement
Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, Zongyuan Ge, and Steven W. Su · 2020
Later among the works it cites.
Overcoming multi-model forgetting in one-shot NAS with diversity maximization
Miao Zhang, Huiqi Li, Shirui Pan, Xiaojun Chang, and Steven W. Su · 2020
Later among the works it cites.
Stage-wise channel pruning for model compression
Original
Mingyang Zhang and Linlin Ou · 2020
Later among the works it cites.
Lambdanetworks: Modeling long-range interactions without attention
Irwan Bello · 2021
Closest in time.
AutoFormer: Searching transformers for visual recognition
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling · 2021
Closest in time.
Exploring simple siamese representation learning
Xinlei Chen and Kaiming He · 2021
Closest in time.
Conditional positional encodings for vision transformers
Original
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen · 2021
Closest in time.
FairNAS: Rethinking evaluation fairness of weight sharing neural architecture search
Xiangxiang Chu, Bo Zhang, and Ruijun Xu · 2021
Closest in time.
NATS-Bench: Benchmarking nas algorithms for architecture topology and size
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys · 2021
Closest in time.
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2021
Closest in time.
Transformer in transformer
Original
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang · 2021
Closest in time.
TransReID: Transformer-based object re-identification
Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang · 2021
Closest in time.
Contrastive embeddings for neural architectures
Original
Daniel Hesslow and Iacopo Poli · 2021
Closest in time.
Transgan: Two transformers can make one strong gan
Original
Yifan Jiang, Shiyu Chang, and Zhangyang Wang · 2021
Closest in time.
Dynamic Slimmable Network
Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang · 2021
Closest in time.
Pi-NAS: Improving neural architecture search by reducing supernet training consistency shift
Jiefeng Peng, Jiqi Zhang, Changlin Li, Guangrun Wang, Xiaodan Liang, and Liang Lin · 2021
Closest in time.
A comprehensive survey of neural architecture search: Challenges and solutions
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang · 2021
Closest in time.
Bottleneck transformers for visual recognition
Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani · 2021
Closest in time.
Training data-efficient image transformers & distillation through attention
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou · 2021
Closest in time.
Tokens-to-token vit: Training vision transformers from scratch on imagenet
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan · 2021
Closest in time.
Semi-supervised blockwisely architecture search for efficient lightweight generative adversarial network
Man Zhang, Yong Zhou, Jiaqi Zhao, Shixiong Xia, Jiaqi Wang, and Zizheng Huang · 2021
Closest in time.
Neural architecture search with random labels
Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun · 2021
Closest in time.
Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al · 2021
Closest in time.
Deformable DETR: Deformable transformers for end-to-end object detection
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…