STAIR: Learning Sparse Text and Image Representation in Grounded Tokens

Understand

Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications.

• State-of-the-art approaches, e.g.
• CLIP, ALIGN, represent images and texts as dense embeddings and calculate the similarity in the dense embedding space as the matching score.
• On the other hand, sparse semantic features like bag-of-words models are more interpretable, but believed to suffer from inferior accuracy than dense representations.