2019

Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains

Meyerson, Elliot, Miikkulainen, Risto

Understand

As deep learning applications continue to become more diverse, an interesting question arises: Can general problem solving arise from jointly learning several such diverse tasks? To approach this question, deep multi-task learning is extended in this paper to the setting where there is no obvious overlap between task architectures.

The idea is that any set of (architecture,task) pairs can be decomposed into a set of potentially related subproblems, whose sharing is optimized by an efficient stochastic algorithm.
The approach is first validated in a classic synthetic multi-task learning benchmark, and then applied to sharing across disparate architectures for vision, NLP, and genomics tasks.
It discovers regularities across these domains, encodes them into sharable modules, and combines these modules systematically to improve performance in the individual tasks.

Built on

A simple weight decay can improve generalization
A. Krogh and J. A. Hertz · 1992
Earlier work this paper cites.
For valid generalization the size of the weights is more important than the size of the network
P. L. Bartlett · 1997
Earlier work this paper cites.
Multitask learning
R. Caruana · 1998
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Introduction to reinforcement learning
R. S. Sutton and A. G. Barto · 1998
Earlier work this paper cites.
Convex multi-task feature learning
A. Argyriou, T. Evgeniou, and M. Pontil · 2008
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning
R. Collobert and J. Weston · 2008
Earlier work this paper cites.
Comparing global and local mutations on bit strings
B. Doerr, T. Jansen, and C. Klein · 2008
Earlier work this paper cites.
On the expectation of the maximum of iid geometric random variables
B. Eisenberg · 2008
Earlier work this paper cites.
Tensor decompositions and applications
T. G. Kolda and B. W. Bader · 2009
Earlier work this paper cites.
Learning Multiple Layers of Features from Tiny Images
A. Krizhevsky · 2009
Earlier work this paper cites.
A hypercube-based encoding for evolving large-scale neural networks
K. O. Stanley, D. B. D’Ambrosio, and J. Gauci · 2009
Earlier work this paper cites.
Learning with whom to share in multi-task feature learning
Z. Kang, K. Grauman, and F. Sha · 2011
Earlier work this paper cites.
Learning task grouping and overlap in multi-task learning
A. Kumar and H. Daumé, III · 2012
Earlier work this paper cites.
Learning to Learn
S. Thrun and L. Pratt · 2012
Earlier work this paper cites.
Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers
J. T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong · 2013
Earlier work this paper cites.
Multi-task learning in deep neural networks for improved phoneme recognition
M. L. Seltzer and J. Droppo · 2013
Earlier work this paper cites.
Tight bounds on the optimization time of a randomized search heuristic on linear functions
C. Witt · 2013
Earlier work this paper cites.
Pac-inspired option discovery in lifelong reinforcement learning
E. Brunskill and L. Li · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2014
Earlier work this paper cites.
Integer and combinatorial optimization
L. A. Wolsey and G. L. Nemhauser · 2014
Earlier work this paper cites.

Similar

Recurrent neural network regularization
Original
W. Zaremba, I. Sutskever, and O. Vinyals · 2014
Cited alongside, same era.
Facial landmark detection by deep multi-task learning
Z. Zhang, L. Ping, L. C. Chen, and T. Xiaoou · 2014
Cited alongside, same era.
Predicting the sequence specificities of dna-and rna-binding proteins by deep learning
B. Alipanahi, A. Delong, M. T. Weirauch, and B. J. Frey · 2015
Cited alongside, same era.
Multi-task learning for multiple language translation
D. Dong, H. Wu, W. He, D. Yu, and H. Wang · 2015
Cited alongside, same era.
Rapid adaptation for deep neural networks through multi-task learning
Z. Huang, J. Li, S. M. Siniscalchi, et al · 2015
Cited alongside, same era.
Representation learning using multi-task deep neural networks for semantic classification and information retrieval
X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y. Y. Wang · 2015
Cited alongside, same era.
On the runtime of randomized local search and simple evolutionary algorithms for dynamic makespan scheduling
F. Neumann and C. Witt · 2015
Cited alongside, same era.
Deep neural networks employing multi-task learning and stacked bottleneck features for speech synthesis
Z. Wu, C. Valentini-Botinhao, O. Watts, and S. King · 2015
Cited alongside, same era.
A unified perspective on multi-domain and multi-task learning
Y. Yang and T. Hospedales · 2015
Cited alongside, same era.
Integrated perception with recurrent multi-task neural networks
H. Bilen and A. Vedaldi · 2016
Cited alongside, same era.
Breaking the billion-variable barrier in real-world optimization using a customized evolutionary algorithm
K. Deb and C. Myburgh · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.

Then

A joint many-task model: Growing a neural network for multiple NLP tasks
K. Hashimoto, C. Xiong, Y. Tsuruoka, and R. Socher · 2017
Later among the works it cites.
Reinforcement learning with unsupervised auxiliary tasks
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu · 2017
Later among the works it cites.
Massively parallel biophysical analysis of crispr-cas complexes on next generation sequencing chips
C. Jung, J. A. Hawkins, S. K. Jones, et al · 2017
Later among the works it cites.
One model to learn them all
Original
L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit · 2017
Later among the works it cites.
Learning multiple tasks with multilinear relationship networks
M. Long, Z. Cao, J. Wang, and P. S. Yu · 2017
Later among the works it cites.
Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification
Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Javidi, and R. S. Feris · 2017
Later among the works it cites.
Automatic differentiation in pytorch
A. Paske et al · 2017
Later among the works it cites.
Learning multiple visual domains with residual adapters
S.-A. Rebuffi, H. Bilen, and A. Vedaldi · 2017
Later among the works it cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean · 2017
Later among the works it cites.
Distral: Robust multitask reinforcement learning
Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu · 2017
Later among the works it cites.
Deep multi-task representation learning: A tensor factorisation approach
Y. Yang and T. Hospedales · 2017
Later among the works it cites.
State abstractions for lifelong reinforcement learning
D. Abel, D. Arumugam, L. Lehnert, and M. Littman · 2018
Later among the works it cites.
Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks
Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich · 2018
Later among the works it cites.
Evolutionary architecture search for deep multitask networks
J. Liang, E. Meyerson, and R. Miikkulainen · 2018
Later among the works it cites.
Regularizing and optimizing LSTM language models
S. Merity, N. S. Keskar, and R. Socher · 2018
Later among the works it cites.
Beyond shared hierarchies: Deep multitask learning through soft layer ordering
E. Meyerson and R. Miikkulainen · 2018
Later among the works it cites.
Pseudo-task augmentation: From deep multitask learning to intratask sharing—and back
E. Meyerson and R. Miikkulainen · 2018
Later among the works it cites.
Efficient parametrization of multi-domain deep neural networks
S.-A. Rebuffi, H. Bilen, and A. Vedaldi · 2018
Later among the works it cites.
Routing networks: Adaptive selection of non-linear functions for multi-task learning
C. Rosenbaum, T. Klinger, and M. Riemer · 2018
Later among the works it cites.
On the robustness of evolutionary algorithms to noise: Refined results and an example where noise helps
D. Sudholt · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…