Finding Visual Task Vectors

Understand

Visual Prompting is a technique for teaching models to perform a visual task via in-context examples, without any additional training.

• In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information.
• Equipped with this insight, we demonstrate that it is possible to identify the task vectors and use them to guide the network towards performing different tasks without providing any input-output examples.
• To find task vectors, we compute the average intermediate activations per task and use the REINFORCE algorithm to search for the subset of task vectors.