Patching open-vocabulary models by interpolating weights

Understand

Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks.

• However, there are still settings where their zero-shot performance is far from optimal.
• We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
• Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched.