Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information

Understand

To effectively exploit the potential of large-scale models, various pre-training strategies supported by massive data from different sources are proposed, including supervised pre-training, weakly-supervised pre-training, and self-supervised pre-training.

• It has been proved that combining multiple pre-training strategies and data from various modalities/sources can greatly boost the training of large-scale models.
• However, current works adopt a multi-stage pre-training system, where the complex pipeline may increase the uncertainty and instability of the pre-training.
• It is thus desirable that these strategies can be integrated in a single-stage manner.