Unified Lexical Representation for Interpretable Visual-Language Alignment

Understand

Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work.

• Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores.
• On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words.
• However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively.