2019

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Sun, Ximeng, Panda, Rameswar, Feris, Rogerio et al.

Understand

Multi-task learning is an open and challenging problem in computer vision.

The typical way of conducting multi-task learning with deep neural networks is either through handcrafted schemes that share all initial layers and branch out at an adhoc point, or through separate task-specific networks with an additional feature sharing/fusion mechanism.
Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks to achieve the best recognition accuracy, while taking resource efficiency into account.
Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network.

Built on

Multi-task learning
R. Caruana · 1997
Earlier work this paper cites.
Learning to learn
S. Thrun and L. Pratt · 1998
Earlier work this paper cites.
Evolving neural networks through augmenting topologies
Kenneth O Stanley and Risto Miikkulainen · 2002
Earlier work this paper cites.
Multi-task learning for classification with dirichlet process priors
Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram · 2007
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston · 2009
Earlier work this paper cites.
Clustered multi-task learning: A convex formulation
Laurent Jacob, Jean-philippe Vert, and Francis R Bach · 2009
Earlier work this paper cites.
Learning with whom to share in multi-task feature learning
Zhuoliang Kang, Kristen Grauman, and Fei Sha · 2011
Earlier work this paper cites.
Clustered multi-task learning via alternating structure optimization
Jiayu Zhou, Jianhui Chen, and Jieping Ye · 2011
Earlier work this paper cites.
Learning task grouping and overlap in multi-task
A. Kumar and H. Daume III · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from rgbd images
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus · 2012
Earlier work this paper cites.
Flexible modeling of latent task structures in multitask learning
Original
Alexandre Passos, Piyush Rai, Jacques Wainer, and Hal Daume III · 2012
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013
Earlier work this paper cites.
Depth map prediction from a single image using a multi-scale deep network
David Eigen, Christian Puhrsch, and Rob Fergus · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
A* sampling
Chris J Maddison, Daniel Tarlow, and Tom Minka · 2014
Earlier work this paper cites.
Conditional computation in neural networks for faster models
Original
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup · 2015
Earlier work this paper cites.
Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture
David Eigen and Rob Fergus · 2015
Earlier work this paper cites.
Cross-domain image retrieval with a dual attribute-aware ranking network
Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan · 2015
Earlier work this paper cites.
Integrated perception with recurrent multi-task neural networks
Hakan Bilen and Andrea Vedaldi · 2016
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Deep networks with stochastic depth
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger · 2016
Earlier work this paper cites.
Categorical reparameterization with gumbel-softmax
Original
Eric Jang, Shixiang Gu, and Ben Poole · 2016
Earlier work this paper cites.
Deep cross residual learning for multi-task visual recognition
B. Jou and S. F. Chang · 2016
Earlier work this paper cites.

Similar

The concrete distribution: A continuous relaxation of discrete random variables
Original
Chris J Maddison, Andriy Mnih, and Yee Whye Teh · 2016
Cited alongside, same era.
Cross-stitch networks for multi-task learning
I. Misra, A. Shrivastava, A. Gupta, and M. Hebert · 2016
Cited alongside, same era.
Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition
Original
R. Ranjan, V. Patel, and R. Chellappa · 2016
Cited alongside, same era.
Progressive neural networks
Original
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell · 2016
Cited alongside, same era.
Residual networks behave like ensembles of relatively shallow networks
Andreas Veit, Michael J Wilber, and Serge Belongie · 2016
Cited alongside, same era.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille · 2017
Cited alongside, same era.
Very deep convolutional networks for text classification
Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun · 2017
Cited alongside, same era.
Blitznet: A real-time deep network for scene understanding
Nikita Dvornik, Konstantin Shmelkov, Julien Mairal, and Cordelia Schmid · 2017
Cited alongside, same era.
Densely connected convolutional networks
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger · 2017
Cited alongside, same era.
Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory
Iasonas Kokkinos · 2017
Cited alongside, same era.
Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogério Schmidt Feris · 2017
Cited alongside, same era.

Then

Blockdrop: Dynamic inference paths in residual networks
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris · 2018
Later among the works it cites.
Taskonomy: Disentangling task transfer learning
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese · 2018
Later among the works it cites.
Learning transferable architectures for scalable image recognition
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le · 2018
Later among the works it cites.
Deep elastic networks with model selection for multi-task learning
Chanho Ahn, Eunwoo Kim, and Songhwai Oh · 2019
Closest in time.
Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels
Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso · 2019
Closest in time.
Neural architecture search: A survey
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter · 2019
Closest in time.
Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille · 2019
Closest in time.
Spottune: transfer learning through adaptive fine-tuning
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris · 2019
Closest in time.
Darts: Differentiable architecture search
Hanxiao Liu, Karen Simonyan, and Yiming Yang · 2019
Closest in time.
End-to-end multi-task learning with attention
Shikun Liu, Edward Johns, and Andrew J Davison · 2019
Closest in time.
Attentive single-tasking of multiple tasks
Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos · 2019
Closest in time.
Moment matching for multi-source domain adaptation
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang · 2019
Closest in time.
Aging evolution for image classifier architecture search
E Real, A Aggarwal, Y Huang, and QV Le · 2019
Closest in time.
Latent multi-task architecture learning
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard · 2019
Closest in time.
Which tasks should be learned together in multi-task learning?
Original
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese · 2019
Closest in time.
Many task learning with task routing
Original
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring · 2019
Closest in time.
Branched multi-task networks: Deciding what layers to share
Original
Simon Vandenhende, Bert De Brabandere, and Luc Van Gool · 2019
Closest in time.
Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer · 2019
Closest in time.
Liteeval: A coarse-to-fine framework for resource efficient video recognition
Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S Davis · 2019
Closest in time.
Snas: stochastic neural architecture search
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin · 2019
Closest in time.
Temporally distributed networks for fast video semantic segmentation
Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff, and Federico Perazzi · 2020
Closest in time.
Real-time semantic segmentation with fast attention
Original
Ping Hu, Federico Perazzi, Fabian Caba Heilbron, Oliver Wang, Zhe Lin, Kate Saenko, and Stan Sclaroff · 2020
Closest in time.
Hierarchical multi-scale attention for semantic segmentation
Original
Andrew Tao, Karan Sapra, and Bryan Catanzaro · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…