Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Understand

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner.

• Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM).
• The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly.
• The last task tries to predict whether an image and a text describe each other.