Composing Ensembles of Pre-trained Models via Iterative Consensus

Understand

Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on.

• Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions.
• In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner.
• We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization.