AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation

Understand

Pre-trained vision-language models (VLMs) have shown impressive results in various visual classification tasks.

• However, we often fail to fully unleash their potential when adapting them for new concept understanding due to limited information on new classes.
• To address this limitation, we introduce a novel adaptation framework, AWT (Augment, Weight, then Transport).
• AWT comprises three key components: augmenting inputs with diverse visual perspectives and enriched class descriptions through image transformations and language models; dynamically weighting inputs based on the prediction entropy; and employing optimal transport to mine semantic correlations in the vision-language space.