Knowledge Composition using Task Vectors with Learned Anisotropic Scaling

Understand

Pre-trained models produce strong generic representations that can be adapted via fine-tuning.

• The learned weight difference relative to the pre-trained model, known as a task vector, characterises the direction and stride of fine-tuning.
• The significance of task vectors is such that simple arithmetic operations on them can be used to combine diverse representations from different domains.
• This paper builds on these properties of task vectors and aims to answer (1) whether components of task vectors, particularly parameter blocks, exhibit similar characteristics, and (2) how such blocks can be used to enhance knowledge composition and transfer.