Grounding Language Models to Images for Multimodal Inputs and Outputs

Understand

We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images.

• Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation.
• We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions.
• This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images.