MoMo: A shared encoder Model for text, image and multi-Modal representations

Understand

We propose a self-supervised shared encoder model that achieves strong results on several visual, language and multimodal benchmarks while being data, memory and run-time efficient.

• We make three key contributions.
• First, in contrast to most existing works, we use a single transformer with all the encoder layers processing both the text and the image modalities.
• Second, we propose a stage-wise training strategy where the model is first trained on images, then jointly with unimodal text and image datasets and finally jointly with text and text-image datasets.