Small Towers Make Big Differences

Understand

Multi-task learning aims at solving multiple machine learning tasks at the same time.

• A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal.
• In this paper, we provide some insights on understanding the trade-off between Pareto efficiency and generalization as a result of parameterization in multi-task deep learning models.
• As a multi-objective optimization problem, enough parameterization is needed for handling task conflicts in a constrained solution space; however, from a multi-task generalization perspective, over-parameterization undermines the benefit of learning a shared representation which helps harder tasks or tasks with limited training examples.