2020

Multi-Task Learning with Deep Neural Networks: A Survey

Crawshaw, Michael

Understand

Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model.

Such approaches offer advantages like improved data efficiency, reduced overfitting through shared representations, and fast learning by leveraging auxiliary information.
However, the simultaneous learning of multiple tasks presents new design and optimization challenges, and choosing which tasks should be learned jointly is in itself a non-trivial problem.
In this survey, we give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.

Built on

Multi-task deep neural networks for natural language understanding
Original
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao · 1901
Earlier work this paper cites.
Improving multi-task deep neural networks via knowledge distillation for natural language understanding
Original
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao · 1904
Earlier work this paper cites.
Learning sparse sharing architectures for multiple tasks
Original
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang · 1911
Earlier work this paper cites.
Adashare: Learning what to share for efficient deep multi-task learning
Original
Ximeng Sun, Rameswar Panda, and Rogerio Feris · 1911
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J. Williams · 1992
Earlier work this paper cites.
Nonlinear multiobjective optimization , volume 12 of International series in operations research and management science
Kaisa Miettinen · 1998
Earlier work this paper cites.
A model of inductive bias learning
Jonathan Baxter · 2000
Earlier work this paper cites.
Accurate unlexicalized parsing
Dan Klein and Christopher D. Manning · 2003
Earlier work this paper cites.
Regularized multi–task learning
Theodoros Evgeniou and Massimiliano Pontil · 2004
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn Treebank
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz · 2004
Earlier work this paper cites.
Model compression
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Earlier work this paper cites.
Pareto front
Johann Dréo · 2006
Earlier work this paper cites.
Convex multi-task feature learning
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil · 2008
Earlier work this paper cites.
A notion of task relatedness yielding provable multiple-task learning guarantees
Shai Ben-David and Reba Schuller Borbely · 2008
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning
Ronan Collobert and Jason Weston · 2008
Earlier work this paper cites.
Representational similarity analysis – connecting the branches of systems neuroscience
Nikolaus Kriegeskorte · 2008
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston · 2009
Earlier work this paper cites.
Taking advantage of sparsity in multi-task learning
Original
Karim Lounici, Massimiliano Pontil, Alexandre B Tsybakov, and Sara Van De Geer · 2009
Earlier work this paper cites.
Natural language processing (almost) from scratch
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa · 2011
Earlier work this paper cites.
Multiple-gradient descent algorithm (mgda) for multiobjective optimization
Jean-Antoine Désidéri · 2012
Earlier work this paper cites.
Learning task grouping and overlap in multi-task learning
Original
Abhishek Kumar and Hal Daume III · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from rgbd images
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Earlier work this paper cites.
Estimating or propagating gradients through stochastic neurons for conditional computation
Original
Yoshua Bengio, Nicholas Léonard, and Aaron Courville · 2013
Earlier work this paper cites.
Ontonotes release 5.0
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Zue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston · 2013
Earlier work this paper cites.
Findings of the 2014 workshop on statistical machine translation
Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna · 2014
Earlier work this paper cites.
A fast and accurate dependency parser using neural networks
Danqi Chen and Christopher Manning · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Dropout: A simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier · 2014
Earlier work this paper cites.
Facial landmark detection by deep multi-task learning
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang · 2014
Earlier work this paper cites.
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh · 2015
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning · 2015
Earlier work this paper cites.
Microsoft coco captions: Data collection and evaluation server
Original
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick · 2015
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David · 2015
Earlier work this paper cites.
Multi-task learning for multiple language translation
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang · 2015
Earlier work this paper cites.
Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Human-level concept learning through probabilistic program induction
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum · 2015
Earlier work this paper cites.
Representation learning using multi-task deep neural networks for semantic classification and information retrieval
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-yi Wang · 2015
Earlier work this paper cites.
Multi-task sequence to sequence learning
Original
Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser · 2015
Earlier work this paper cites.
Actor-mimic: Deep multitask and transfer reinforcement learning
Original
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov · 2015
Earlier work this paper cites.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik · 2015
Earlier work this paper cites.
Policy distillation
Original
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell · 2015
Earlier work this paper cites.
End-to-end memory networks
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al · 2015
Earlier work this paper cites.
Multi-task learning and algorithmic stability
Yu Zhang · 2015
Earlier work this paper cites.
When is multitask learning effective? semantic sequence prediction under varying data conditions
Original
Héctor Martínez Alonso and Barbara Plank · 2016
Earlier work this paper cites.
Neural module networks
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein · 2016
Earlier work this paper cites.
Deepmind lab
Original
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al · 2016
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Instance-aware semantic segmentation via multi-task network cascades
Jifeng Dai, Kaiming He, and Jian Sun · 2016
Earlier work this paper cites.
A joint many-task model: Growing a neural network for multiple nlp tasks
Original
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Learning and transfer of modulated locomotor controllers
Original
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver · 2016
Earlier work this paper cites.
Reinforcement learning with unsupervised auxiliary tasks
Original
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.

Similar

Categorical reparameterization with gumbel-softmax
Original
Eric Jang, Shixiang Gu, and Ben Poole · 2016
Cited alongside, same era.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum · 2016
Cited alongside, same era.
Asymmetric multi-task learning based on task relatedness and loss
Giwoong Lee, Eunho Yang, and Sung Hwang · 2016
Cited alongside, same era.
Self-paced multi-task learning
Original
Changsheng Li, Junchi Yan, Fan Wei, Weishan Dong, Qingshan Liu, and Hongyuan Zha · 2016
Cited alongside, same era.
Cross-stitch networks for multi-task learning
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert · 2016
Cited alongside, same era.
Learning values across many orders of magnitude
Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver · 2016
Cited alongside, same era.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V Le · 2016
Cited alongside, same era.
Modular multitask reinforcement learning with policy sketches
Jacob Andreas, Dan Klein, and Sergey Levine · 2017
Cited alongside, same era.
Identifying beneficial task relations for multi-task learning in deep neural networks
Original
Joachim Bingel and Anders Søgaard · 2017
Cited alongside, same era.
Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, 2017
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich · 2017
Cited alongside, same era.
Guesswhat?! visual object discovery through multi-modal dialogue
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville · 2017
Cited alongside, same era.
Learning modular neural network policies for multi-task and multi-robot transfer
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.

Then

Efficient parametrization of multi-domain deep neural networks
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi · 2018
Later among the works it cites.
Multi-task learning as multi-objective optimization
Ozan Sener and Vladlen Koltun · 2018
Later among the works it cites.
Gradient adversarial training of neural networks
Original
Ayan Sinha, Zhao Chen, Vijay Badrinarayanan, and Andrew Rabinovich · 2018
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Original
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman · 2018
Later among the works it cites.
Taskonomy: Disentangling task transfer learning
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese · 2018
Later among the works it cites.
Learning synergies between pushing and grasping with self-supervised deep reinforcement learning
Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas Funkhouser · 2018
Later among the works it cites.
A modulation module for multi-task learning with applications in image retrieval
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu · 2018
Later among the works it cites.
Pyramidal person re-identification via multi-loss dynamic training, 2018
Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji · 2018
Later among the works it cites.
Task2vec: Task embedding for meta-learning
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona · 2019
Later among the works it cites.
Deep elastic networks with model selection for multi-task learning
Chanho Ahn, Eunwoo Kim, and Songhwai Oh · 2019
Later among the works it cites.
Multi-task learning for multi-modal emotion recognition and sentiment analysis
Original
Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya · 2019
Later among the works it cites.
Solving rubik’s cube with a robot hand
Original
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al · 2019
Later among the works it cites.
Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels
Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso · 2019
Later among the works it cites.
Recursive routing networks: Learning to compose modules for language understanding
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, and Lauri Karttunen · 2019
Later among the works it cites.
Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning, 2019
Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh · 2019
Later among the works it cites.
Bam! born-again multi-task networks for natural language understanding
Original
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le · 2019
Later among the works it cites.
Representation similarity analysis for efficient task taxonomy & transfer learning
Kshitij Dwivedi and Gemma Roig · 2019
Later among the works it cites.
Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille · 2019
Later among the works it cites.
A comparison of loss weighting strategies for multi task learning in deep neural networks, 2019
Ting Gong, Tyler Lee, Cory Stephenson, Venkata Renduchintala, Suchismita Padhy, Anthony Ndirango, Gokce Keskin, and Oguz H. Elibol · 2019
Later among the works it cites.
Gqa: A new dataset for real-world visual reasoning and compositional question answering
Drew A Hudson and Christopher D Manning · 2019
Later among the works it cites.
Adaptive scheduling for multi-task learning
Original
Sébastien Jean, Orhan Firat, and Melvin Johnson · 2019
Later among the works it cites.
Meta reinforcement learning with task embedding and shared policy
Original
Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang · 2019
Later among the works it cites.
Pareto multi-task learning
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong · 2019
Later among the works it cites.
End-to-end multi-task learning with attention
S. Liu, E. Johns, and A. J. Davison · 2019
Later among the works it cites.
Attentive single-tasking of multiple tasks
Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos · 2019
Later among the works it cites.
Evolving deep neural networks
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al · 2019
Later among the works it cites.
Generalization in multitask deep neural classifiers: a statistical physics approach
Anthony Ndirango and Tyler Lee · 2019
Later among the works it cites.
Feature partitioning for efficient multi-task architectures
Original
Alejandro Newell, Lu Jiang, Chong Wang, Li-Jia Li, and Jia Deng · 2019
Later among the works it cites.
Multi-task learning of hierarchical vision-language representation
Duy-Kien Nguyen and Takayuki Okatani · 2019
Later among the works it cites.
Continual lifelong learning with neural networks: A review
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter · 2019
Later among the works it cites.
Omninet: A unified architecture for multi-modal multi-task learning
Original
Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain · 2019
Later among the works it cites.
Diversity and depth in per-example routing models
Prajit Ramachandran and Quoc V. Le · 2019
Later among the works it cites.
Routing networks and the challenges of modular and compositional computation
Original
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger · 2019
Later among the works it cites.
Latent multi-task architecture learning
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard · 2019
Later among the works it cites.
A hierarchical multi-task approach for learning embeddings from semantic tasks
Victor Sanh, Thomas Wolf, and Sebastian Ruder · 2019
Later among the works it cites.
A principled approach for learning task similarity in multitask learning
Original
Changjian Shui, Mahdieh Abbasi, Louis-Émile Robitaille, Boyu Wang, and Christian Gagné · 2019
Later among the works it cites.
Deep model transferability from attribution maps
Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song · 2019
Later among the works it cites.
Which tasks should be learned together in multi-task learning?
Original
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese · 2019
Later among the works it cites.
Learning task relatedness in multi-task learning for images in context
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring · 2019
Later among the works it cites.
Branched multi-task networks: deciding what layers to share
Original
Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool · 2019
Later among the works it cites.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
Original
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine · 2019
Later among the works it cites.
Pattern-affinitive propagation across depth, surface normal and semantic segmentation
Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang · 2019
Later among the works it cites.
A comprehensive survey on transfer learning
Original
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He · 2019
Later among the works it cites.
Learning functions to study the benefit of multitask learning
Original
Gabriele Bettgenhäuser, Michael A Hedderich, and Dietrich Klakow · 2020
Closest in time.
Sharing knowledge in multi-task deep reinforcement learning
Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters · 2020
Closest in time.
Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning
Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu · 2020
Closest in time.
Meta-learning in neural networks: A survey
Original
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey · 2020
Closest in time.
12-in-1: Multi-task vision and language representation learning
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee · 2020
Closest in time.
Maximum roaming multi-task learning
Original
Lucas Pascal, Pietro Michiardi, Xavier Bost, Benoit Huet, and Maria A Zuluaga · 2020
Closest in time.
Mti-net: Multi-scale task interaction networks for multi-task learning
Original
Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool · 2020
Closest in time.
Understanding and improving information transfer in multi-task learning
Original
Sen Wu, Hongyang R Zhang, and Christopher Ré · 2020
Closest in time.
Multi-task reinforcement learning with soft modularization
Original
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang · 2020
Closest in time.
Gradient surgery for multi-task learning
Original
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn · 2020
Closest in time.
Multi-task reinforcement learning as a hidden-parameter block mdp
Original
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau · 2020
Closest in time.
Deep multi-task learning with low level tasks supervised at lower layers
Anders Søgaard and Yoav Goldberg · 2038
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…