Language Models Can See: Plugging Visual Controls in Text Generation

Understand

Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality.

• While they are designed for text-prompted generation, it remains an open question how the generation process could be guided by modalities beyond text such as images.
• In this work, we propose a training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP), for plugging in visual controls in the generation process and enabling LMs to perform multimodal tasks (e.g., image captioning) in a zero-shot manner.
• MAGIC is a simple yet efficient plug-and-play framework, which directly combines an off-the-shelf LM (i.e., GPT-2) and an image-text matching model (i.e., CLIP) for image-grounded text generation.