DPL: Decoupled Prompt Learning for Vision-Language Models

Understand

Prompt learning has emerged as an efficient and effective approach for transferring foundational Vision-Language Models (e.g., CLIP) to downstream tasks.

• However, current methods tend to overfit to seen categories, thereby limiting their generalization ability for unseen classes.
• In this paper, we propose a new method, Decoupled Prompt Learning (DPL), which reformulates the attention in prompt learning to alleviate this problem.
• Specifically, we theoretically investigate the collaborative process between prompts and instances (i.e., image patches/text tokens) by reformulating the original self-attention into four separate sub-processes.