Emu: Generative Pretraining in Multimodality

Understand

We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context.

• This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process.
• First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence.
• Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence.