Data Efficient Masked Language Modeling for Vision and Language

Understand

Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining.

• In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text.
• In this paper, we observe several key disadvantages of MLM in this setting.
• First, as captions tend to be short, in a third of the sentences no token is sampled.