Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining

Understand

Recent work in vision-and-language pretraining has investigated supervised signals from object detection data to learn better, fine-grained multimodal representations.

• In this work, we take a step further and explore how we can tap into supervision from small-scale visual relation data.
• In particular, we propose two pretraining approaches to contextualise visual entities in a multimodal setup.
• With verbalised scene graphs, we transform visual relation triplets into structured captions, and treat them as additional image descriptions.