What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases

Understand

Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate.

• A common belief is that a small number of VL skills underlie the variety of VL tests.
• In this paper, we perform a large-scale transfer learning experiment aimed at discovering latent VL skills from data.
• We reveal interesting characteristics that have important implications for test suite design.