Diagnosing and Rectifying Vision Models using Language

Understand

Recent multi-modal contrastive learning models have demonstrated the ability to learn an embedding space suitable for building strong vision classifiers, by leveraging the rich information in large-scale image-caption datasets.

• Our work highlights a distinct advantage of this multi-modal embedding space: the ability to diagnose vision classifiers through natural language.
• The traditional process of diagnosing model behaviors in deployment settings involves labor-intensive data acquisition and annotation.
• Our proposed method can discover high-error data slices, identify influential attributes and further rectify undesirable model behaviors, without requiring any visual data.