2024

No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

Pouget, Angéline, Beyer, Lucas, Bugliarello, Emanuele et al.

Understand

We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs).

Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings.
First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding.
Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets.

Built on

The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2020) · 1981
Earlier work this paper cites.
Are we done with imagenet?
Original
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., and Oord, A. v. d. (2020) · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009) · 2009
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · 2010
Earlier work this paper cites.
mt5: A massively multilingual pre-trained text-to-text transformer
Original
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020) · 2010
Earlier work this paper cites.
Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014) · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. (2014) · 2014
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · 2015
Earlier work this paper cites.
In-datacenter performance analysis of a tensor processing unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D. A., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark, C., Coriell, J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland, W., Hagmann, R., Ho, C. R., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey, A., Jaworski, A., Kaplan, A., Khaitan, H., Koch, A., Kumar, N., Lacy, S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G., Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix, K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Salek, A., Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A., Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang, W., Wilcox, E., and Yoon, D. H. (2017) · 2017
Earlier work this paper cites.
No classification without representation: Assessing geodiversity issues in open data sets for the developing world
Original
Shankar, S., Halpern, Y., Breck, E., Atwood, J., Wilson, J., and Sculley, D. (2017) · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017) · 2017
Earlier work this paper cites.
Jax: composable transformations of python+ numpy programs
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., et al. (2018) · 2018
Earlier work this paper cites.
Adafactor: Adaptive learning rates with sublinear memory cost
Shazeer, N. and Stern, M. (2018) · 2018
Earlier work this paper cites.
Does object recognition work for everyone?
De Vries, T., Misra, I., Wang, C., and Van der Maaten, L. (2019) · 2019
Earlier work this paper cites.
Model cards for model reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T. (2019) · 2019
Earlier work this paper cites.
Flax: A neural network library and ecosystem for jax
Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. (2020) · 2020
Earlier work this paper cites.
Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval
Weyand, T., Araujo, A., Cao, B., and Sim, J. (2020) · 2020
Earlier work this paper cites.
Evaluating clip: towards characterization of broader capabilities and downstream implications
Original
Agarwal, S., Krueger, G., Clark, J., Radford, A., Kim, J. W., and Brundage, M. (2021) · 2021
Earlier work this paper cites.
Multimodal datasets: misogyny, pornography, and malignant stereotypes
Original
Birhane, A., Prabhu, V. U., and Kahembwe, E. (2021) · 2021
Earlier work this paper cites.
Systematic inequalities in language technology performance across the world’s languages
Original
Blasi, D., Anastasopoulos, A., and Neubig, G. (2021) · 2021
Earlier work this paper cites.
Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts
Bugliarello, E., Cotterell, R., Okazaki, N., and Elliott, D. (2021) · 2021
Earlier work this paper cites.
Adaptive methods for real-world domain generalization
Dubey, A., Ramanathan, V., Pentland, A., and Mahajan, D. (2021) · 2021
Earlier work this paper cites.
Natural adversarial examples
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. (2021) · 2021
Earlier work this paper cites.
Scaling up visual and vision-language representation learning with noisy text supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. (2021) · 2021
Earlier work this paper cites.

Similar

Towards a fairer landmark recognition dataset
Original
Kim, Z., Araujo, A., Cao, B., Askew, C., Sim, J., Green, M., Yilla, N., and Weyand, T. (2021) · 2021
Cited alongside, same era.
Visually grounded reasoning across languages and cultures
Liu, F., Bugliarello, E., Ponti, E. M., Reddy, S., Collier, N., and Elliott, D. (2021) · 2021
Cited alongside, same era.
xgqa: Cross-lingual visual question answering
Original
Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M. O., Roth, S., Vulić, I., and Gurevych, I. (2021) · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · 2021
Cited alongside, same era.
Broaden the vision: Geo-diverse visual commonsense reasoning
Original
Yin, D., Li, L. H., Hu, Z., Peng, N., and Chang, K.-W. (2021) · 2021
Cited alongside, same era.
Florence: A new foundation model for computer vision
Original
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · 2021
Cited alongside, same era.
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. (2022) · 2022
Cited alongside, same era.
Big vision
Beyer, L., Zhai, X., and Kolesnikov, A. (2022) · 2022
Cited alongside, same era.
IGLUE: A benchmark for transfer learning across modalities, tasks, and languages
Bugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., and Vuli’c, I. (2022) · 2022
Cited alongside, same era.
Towards multi-lingual visual question answering
Original
Changpinyo, S., Xue, L., Szpektor, I., Thapliyal, A. V., Amelot, J., Chen, X., and Soricut, R. (2022) · 2022
Cited alongside, same era.
Pali: A jointly-scaled multilingual language-image model
Original
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · 2022
Cited alongside, same era.
Fairness indicators for systematic assessments of visual feature extractors
Goyal, P., Soriano, A. R., Hazirbas, C., Sagun, L., and Usunier, N. (2022b) · 2022
Cited alongside, same era.

Then

Clip the bias: How useful is balancing data in multimodal learning?
Alabdulmohsin, I., Wang, X., Steiner, A. P., Goyal, P., D’Amour, A., and Zhai, X. (2023) · 2023
Later among the works it cites.
Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023) · 2023
Later among the works it cites.
Evaluating bias and fairness in gender-neutral pretrained vision-and-language models
Cabello, L., Bugliarello, E., Brandl, S., and Elliott, D. (2023) · 2023
Later among the works it cites.
Pali-3 vision language models: Smaller, faster, stronger
Chen, X., Wang, X., Beyer, L., Kolesnikov, A., Wu, J., Voigtlaender, P., Mustafa, B., Goodman, S., Alabdulmohsin, I., Padlewski, P., Salz, D., Xiong, X., Vlasic, D., Pavetic, F., Rong, K., Yu, T., Keysers, D., Zhai, X., and Soricut, R. (2023) · 2023
Later among the works it cites.
Do multilingual language models think better in english?
Original
Etxaniz, J., Azkune, G., Soroa, A., de Lacalle, O. L., and Artetxe, M. (2023) · 2023
Later among the works it cites.
Data filtering networks
Original
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. (2023) · 2023
Later among the works it cites.
Uncurated image-text datasets: Shedding light on demographic bias
Garcia, N., Hirota, Y., Wu, Y., and Nakashima, Y. (2023) · 2023
Later among the works it cites.
Vision-language models performing zero-shot tasks exhibit gender-based disparities
Original
Hall, M., Gustafson, L., Adcock, A., Misra, I., and Ross, C. (2023) · 2023
Later among the works it cites.
Geonet: Benchmarking unsupervised adaptation across geographies
Kalluri, T., Xu, W., and Chandraker, M. (2023) · 2023
Later among the works it cites.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models
Li, J., Li, D., Savarese, S., and Hoi, S. (2023) · 2023
Later among the works it cites.
Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. (2023) · 2023
Later among the works it cites.
Bridging the digital divide: Performance variation across socio-economic factors in vision-language models
Original
Nwatu, J., Ignat, O., and Mihalcea, R. (2023) · 2023
Later among the works it cites.
Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets
Original
Smith, B., Farinha, M., Hall, S. M., Kirk, H. R., Shtedritski, A., and Bain, M. (2023) · 2023
Later among the works it cites.
EVA-CLIP: improved training techniques for CLIP at scale
Original
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. (2023) · 2023
Later among the works it cites.
Image captioners are scalable vision learners too
Tschannen, M., Kumar, M., Steiner, A., Zhai, X., Houlsby, N., and Beyer, L. (2023) · 2023
Later among the works it cites.
Sigmoid loss for language image pre-training
Original
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023) · 2023
Later among the works it cites.
Large-scale domain-specific pretraining for biomedical vision-language processing
Original
Zhang, S., Xu, Y., Usuyama, N., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., Wong, C., et al. (2023) · 2023
Later among the works it cites.
Datacomp: In search of the next generation of multimodal datasets
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. (2024) · 2024
Closest in time.
Geode: a geographically diverse evaluation dataset for object recognition
Ramaswamy, V. V., Lin, S. Y., Zhao, D., Adcock, A., van der Maaten, L., Ghadiyaram, D., and Russakovsky, O. (2024) · 2024
Closest in time.
Does progress on object recognition benchmarks improve real-world generalization?
Richards, M., Kirichenko, P., Bouchacourt, D., and Ibrahim, M. (2024) · 2024
Closest in time.
EVA-CLIP-18B: scaling CLIP to 18 billion parameters
Original
Sun, Q., Wang, J., Yu, Q., Cui, Y., Zhang, F., Zhang, X., and Wang, X. (2024) · 2024
Closest in time.
Locca: Visual pretraining with location-aware captioners
Wan, B., Tschannen, M., Xian, Y., Pavetic, F., Alabdulmohsin, I., Wang, X., Pinto, A. S., Steiner, A., Beyer, L., and Zhai, X. (2024) · 2024
Closest in time.
Rsud20k: A dataset for road scene understanding in autonomous driving
Original
Zunair, H., Khan, S., and Hamza, A. B. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…