2020

Understanding and Improving Information Transfer in Multi-Task Learning

Wu, Sen, Zhang, Hongyang R., Ré, Christopher

Understand

We investigate multi-task learning approaches that use a shared feature representation for all tasks.

To better understand the transfer of task information, we study an architecture with a shared module for all tasks and a separate output module for each task.
We study the theory of this setting on linear and ReLU-activated models.
Our key observation is that whether or not tasks' data are well-aligned can significantly affect the performance of multi-task learning.

Built on

Multi-task deep neural networks for natural language understanding
Original
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao · 1901
Earlier work this paper cites.
Bridging theory and algorithm for domain adaptation
Original
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I Jordan · 1904
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 1907
Earlier work this paper cites.
A model of inductive bias learning
Jonathan Baxter · 2000
Earlier work this paper cites.
Learning question classifiers
Xin Li and Dan Roth · 2002
Earlier work this paper cites.
Exploiting task relatedness for multiple task learning
Shai Ben-David and Reba Schuller · 2003
Earlier work this paper cites.
Regularized multi-task learning
Theodoros Evgeniou and Massimiliano Pontil · 2004
Earlier work this paper cites.
Mining and summarizing customer reviews
Minqing Hu and Bing Liu · 2004
Earlier work this paper cites.
A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts
Bo Pang and Lillian Lee · 2004
Earlier work this paper cites.
A framework for learning predictive structures from multiple tasks and unlabeled data
Rie Kubota Ando and Tong Zhang · 2005
Earlier work this paper cites.
The elements of statistical learning: data mining, inference and prediction
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and James Franklin · 2005
Earlier work this paper cites.
Kernels for multi–task learning
Charles A Micchelli and Massimiliano Pontil · 2005
Earlier work this paper cites.
Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
Bo Pang and Lillian Lee · 2005
Earlier work this paper cites.
Annotating expressions of opinions and emotions in language
Janyce Wiebe, Theresa Wilson, and Claire Cardie · 2005
Earlier work this paper cites.
Domain adaptation with structural correspondence learning
John Blitzer, Ryan McDonald, and Fernando Pereira · 2006
Earlier work this paper cites.
Bounds for linear multi-task learning
Andreas Maurer · 2006
Earlier work this paper cites.
Multi-task learning for classification with dirichlet process priors
Ya Xue, Xuejun Liao, Lawrence Carin, and Balaji Krishnapuram · 2007
Earlier work this paper cites.
An algorithm for transfer learning in a heterogeneous environment
Andreas Argyriou, Andreas Maurer, and Massimiliano Pontil · 2008
Earlier work this paper cites.
Transfer learning using kolmogorov complexity: Basic theory and empirical evaluations
MM Mahmud and Sylvian Ray · 2008
Earlier work this paper cites.
Distant supervision for relation extraction without labeled data
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky · 2009
Earlier work this paper cites.
A survey on transfer learning
Sinno Jialin Pan and Qiang Yang · 2009
Earlier work this paper cites.

Similar

A theory of learning from different domains
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan · 2010
Cited alongside, same era.
Joint covariate selection and joint subspace selection for multiple classification problems
Guillaume Obozinski, Ben Taskar, and Michael I Jordan · 2010
Cited alongside, same era.
Unsupervised visual domain adaptation using subspace alignment
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars · 2013
Cited alongside, same era.
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts · 2013
Cited alongside, same era.
Convolutional neural networks for sentence classification
Original
Yoon Kim · 2014
Cited alongside, same era.
A regularization approach to learning task relationships in multitask learning
Yu Zhang and Dit-Yan Yeung · 2014
Cited alongside, same era.
Dissimilarity-based sparse subset selection
Ehsan Elhamifar, Guillermo Sapiro, and S Shankar Sastry · 2015
Cited alongside, same era.
Multi-task and lifelong learning of kernels
Anastasia Pentina and Shai Ben-David · 2015
Cited alongside, same era.
Curriculum learning of multiple tasks
Anastasia Pentina, Viktoriia Sharmanska, and Christoph H Lampert · 2015
Cited alongside, same era.
An introduction to matrix concentration inequalities
Joel A Tropp et al · 2015
Cited alongside, same era.
Multi-task learning for subspace segmentation
Yu Wang, David Wipf, Qing Ling, Wei Chen, and Ian James Wassell · 2015
Cited alongside, same era.
When is multitask learning effective? semantic sequence prediction under varying data conditions
Original
Héctor Martínez Alonso and Barbara Plank · 2016
Cited alongside, same era.

Then

A survey on multi-task learning
Original
Yu Zhang and Qiang Yang · 2017
Later among the works it cites.
Matrix completion and related problems via strong duality
Maria-Florina Balcan, Yingyu Liang, David P Woodruff, and Hongyang Zhang · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Later among the works it cites.
Multi-task learning using uncertainty to weigh losses for scene geometry and semantics
Alex Kendall, Yarin Gal, and Roberto Cipolla · 2018
Later among the works it cites.
An introduction to domain adaptation and transfer learning
Original
Wouter M Kouw · 2018
Later among the works it cites.
Simple recurrent units for highly parallelizable recurrence
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi · 2018
Later among the works it cites.
Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang · 2018
Later among the works it cites.
The computational complexity of training relu (s)
Original
Pasin Manurangsi and Daniel Reichman · 2018
Later among the works it cites.
The natural language decathlon: Multitask learning as question answering
Original
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher · 2018
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman · 2018
Later among the works it cites.
Taskonomy: Disentangling task transfer learning
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese · 2018
Later among the works it cites.
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal · 2019
Later among the works it cites.
Autosem: Automatic task selection and mixing in multi-task learning
Original
Han Guo, Ramakanth Pasunuru, and Mohit Bansal · 2019
Later among the works it cites.
Provable guarantees for gradient-based meta-learning
Original
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar · 2019
Later among the works it cites.
Efficient multi-domain learning by covariance normalization
Yunsheng Li and Nuno Vasconcelos · 2019
Later among the works it cites.
A principled approach for learning task similarity in multitask learning
Original
Changjian Shui, Mahdieh Abbasi, Louis-Émile Robitaille, Boyu Wang, and Christian Gagné · 2019
Later among the works it cites.
Which tasks should be learned together in multi-task learning?
Original
Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese · 2019
Later among the works it cites.
Benign overfitting in linear regression
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler · 2020
Closest in time.
Few-shot learning via learning the representation, provably
Original
Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei · 2020
Closest in time.
Meta-learning for mixed linear regression
Original
Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, and Sewoong Oh · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…