ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles

Understand

Automatically generating textual content with desired attributes is an ambitious task that people have pursued long.

• Existing works have made a series of progress in incorporating unimodal controls into language models (LMs), whereas how to generate controllable sentences with multimodal signals and high efficiency remains an open question.
• To tackle the puzzle, we propose a new paradigm of zero-shot controllable text generation with multimodal signals (\textsc{ZeroGen}).
• Specifically, \textsc{ZeroGen} leverages controls of text and image successively from token-level to sentence-level and maps them into a unified probability space at decoding, which customizes the LM outputs by weighted addition without extra training.