Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

Understand

Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks.

• In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance.
• In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning.
• We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups.