HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs

Understand

The hubness problem widely exists in high-dimensional embedding space and is a fundamental source of error for cross-modal matching tasks.

• In this work, we study the emergence of hubs in Visual Semantic Embeddings (VSE) with application to text-image matching.
• We analyze the pros and cons of two widely adopted optimization objectives for training VSE and propose a novel hubness-aware loss function (HAL) that addresses previous methods' defects.
• Unlike (Faghri et al.2018) which simply takes the hardest sample within a mini-batch, HAL takes all samples into account, using both local and global statistics to scale up the weights of "hubs".