Vision Transformer Adapters for Generalizable Multitask Learning

Understand

We introduce the first multitasking vision transformer adapters that learn generalizable task affinities which can be applied to novel tasks and domains.

• Integrated into an off-the-shelf vision transformer backbone, our adapters can simultaneously solve multiple dense vision tasks in a parameter-efficient manner, unlike existing multitasking transformers that are parametrically expensive.
• In contrast to concurrent methods, we do not require retraining or fine-tuning whenever a new task or domain is added.
• We introduce a task-adapted attention mechanism within our adapter framework that combines gradient-based task similarities with attention-based ones.