CLIP4IDC: CLIP for Image Difference Captioning

Understand

Image Difference Captioning (IDC) aims at generating sentences to describe differences between two similar-looking images.

• Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature extractor.
• Accordingly, two major issues may arise: (1) a large domain gap usually exists between the pre-training datasets used for training such a visual encoder and that of the downstream IDC task, and (2) the visual feature extractor, when separately encoding two images, often does not effectively encode the visual changes between two images.
• Due to the excellent zero-shot performance of the recently proposed CLIP, we thus propose CLIP4IDC to transfer a CLIP model for the IDC task to address those issues.