FETA: Towards Specializing Foundation Models for Expert Task Applications

Understand

Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization.

• However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e.g.
• retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training.
• This underlines the necessity to explicitly evaluate and finetune FMs on such expert tasks, arguably ones that appear the most in practical real-world applications.