Language Models are General-Purpose Interfaces

Understand

Foundation models have received much attention due to their effectiveness across a broad range of downstream applications.

• Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities.
• In this work, we propose to use language models as a general-purpose interface to various foundation models.
• A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer.