M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance

Understand

We present M2-omni, a cutting-edge, open-source omni-MLLM that achieves competitive performance to GPT-4o.

• M2-omni employs a unified multimodal sequence modeling framework, which empowers Large Language Models(LLMs) to acquire comprehensive cross-modal understanding and generation capabilities.
• Specifically, M2-omni can process arbitrary combinations of audio, video, image, and text modalities as input, generating multimodal sequences interleaving with audio, image, or text outputs, thereby enabling an advanced and interactive real-time experience.
• The training of such an omni-MLLM is challenged by significant disparities in data quantity and convergence rates across modalities.