Visually Grounded Reasoning across Languages and Cultures

Understand

The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet.

• While one can hardly overestimate how much this benchmark contributed to progress in computer vision, it is mostly derived from lexical databases and image queries in English, resulting in source material with a North American or Western European bias.
• Therefore, we devise a new protocol to construct an ImageNet-style hierarchy representative of more languages and cultures.
• In particular, we let the selection of both concepts and images be entirely driven by native speakers, rather than scraping them automatically.