ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Understand

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences.

• While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image.
• In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning steps.
• This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models.