Built on

S. Sheng, A. N. Venkitasubramanian, and M.-F. Moens, “A Markov network based passage retrieval method for multimodal question answering in the cultural heritage domain,” in Lecture notes in computer science, 2018, pp. 3–15
2018
Earlier work this paper cites.
E. Cetinic, “Iconographic image captioning for artworks,” arXiv.org, Feb. 07, 2021. https://arxiv.org/abs/2102.03942
Original
2021
Earlier work this paper cites.
J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” arXiv.org, Apr. 29, 2022. https://arxiv.org/abs/2204.14198
Original
2022
Earlier work this paper cites.
O. Burda-Lassen, “Ukrainian-To-English Folktale Corpus: Parallel Corpus Creation and augmentation for Machine Translation in Low-Resource Languages,” ACL Anthology, Sep. 01, 2022. https://aclanthology.org/2022.amta-coco4mt.4/
2022
Earlier work this paper cites.
J. Wang et al., “GIT: a generative image-to-text transformer for vision and language,” arXiv.org, May 27, 2022. https://arxiv.org/abs/2205.14100
Original
2022
Earlier work this paper cites.
J. Achiam et al., “GPT-4 Technical Report,” arXiv.org, Mar. 15, 2023. https://arxiv.org/abs/2303.08774
Original
2023
Earlier work this paper cites.
R. Anil et al., “Gemini: a family of highly capable multimodal models,” arXiv.org, Dec. 19, 2023. https://arxiv.org/abs/2312.11805
Original
2023
Earlier work this paper cites.
A. Awadalla et al., “OpenFlamingo: an Open-Source framework for training large autoregressive Vision-Language models,” arXiv.org, Aug. 02, 2023. https://arxiv.org/abs/2308.01390
Original
2023
Earlier work this paper cites.

Similar

S. Bai et al., “TouchStone: Evaluating Vision-Language Models by Language Models” arXiv.org, Aug. 31, 2023. https://arxiv.org/abs/2308.16890
Original
2023
Cited alongside, same era.
O. Burda-Lassen, “Machine Translation of Folktales: small-data-driven and LLM-based approaches,” ACL Anthology, Sep. 01, 2023. https://aclanthology.org/2023.clasp-1.8/
2023
Cited alongside, same era.
W. Dai et al., “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” arXiv.org, May 11, 2023. https://arxiv.org/abs/2305.06500
Original
2023
Cited alongside, same era.
C. Fu et al., “MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language models” arXiv.org, Jun. 23, 2023. https://arxiv.org/abs/2306.13394
Original
2023
Cited alongside, same era.

Then

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv.org, Apr. 17, 2023. https://arxiv.org/abs/2304.08485
Original
2023
Later among the works it cites.
S. Yin et al., “Woodpecker: Hallucination Correction for Multimodal large Language models,” arXiv.org, Oct. 24, 2023. https://arxiv.org/abs/2310.16045
Original
2023
Later among the works it cites.
B. Zheng et al., “Image captioning for cultural artworks: a case study on ceramics,” Multimedia Systems, vol. 29, no. 6, pp. 3223–3243, Sep. 2023, doi: 10.1007/s00530-023-01178-8
2023
Later among the works it cites.
Y. Cao, W. Li, J. Li, Y. Yuan, A. Karamolegkou, and D. Hershcovich, “Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing,” arXiv.org, Feb. 08, 2024. https://arxiv.org/abs/2402.06015
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

How Culturally Aware are Vision-Language Models?

Burda-Lassen, Olena, Chadha, Aman, Goswami, Shashank et al.

Understand

An image is often considered worth a thousand words, and certain images can tell rich and insightful stories.

Can these stories be told via image captioning? Images from folklore genres, such as mythology, folk dance, cultural signs, and symbols, are vital to every culture.
Our research compares the performance of four popular vision-language models (GPT-4V, Gemini Pro Vision, LLaVA, and OpenFlamingo) in identifying culturally specific information in such images and creating accurate and culturally sensitive image captions.
We also propose a new evaluation metric, the Cultural Awareness Score (CAS), which measures the degree of cultural awareness in image captions.

Built on

S. Sheng, A. N. Venkitasubramanian, and M.-F. Moens, “A Markov network based passage retrieval method for multimodal question answering in the cultural heritage domain,” in Lecture notes in computer science, 2018, pp. 3–15
2018
Earlier work this paper cites.
E. Cetinic, “Iconographic image captioning for artworks,” arXiv.org, Feb. 07, 2021. https://arxiv.org/abs/2102.03942
Original
2021
Earlier work this paper cites.
J.-B. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” arXiv.org, Apr. 29, 2022. https://arxiv.org/abs/2204.14198
Original
2022
Earlier work this paper cites.
O. Burda-Lassen, “Ukrainian-To-English Folktale Corpus: Parallel Corpus Creation and augmentation for Machine Translation in Low-Resource Languages,” ACL Anthology, Sep. 01, 2022. https://aclanthology.org/2022.amta-coco4mt.4/
2022
Earlier work this paper cites.
J. Wang et al., “GIT: a generative image-to-text transformer for vision and language,” arXiv.org, May 27, 2022. https://arxiv.org/abs/2205.14100
Original
2022
Earlier work this paper cites.
J. Achiam et al., “GPT-4 Technical Report,” arXiv.org, Mar. 15, 2023. https://arxiv.org/abs/2303.08774
Original
2023
Earlier work this paper cites.
R. Anil et al., “Gemini: a family of highly capable multimodal models,” arXiv.org, Dec. 19, 2023. https://arxiv.org/abs/2312.11805
Original
2023
Earlier work this paper cites.
A. Awadalla et al., “OpenFlamingo: an Open-Source framework for training large autoregressive Vision-Language models,” arXiv.org, Aug. 02, 2023. https://arxiv.org/abs/2308.01390
Original
2023
Earlier work this paper cites.

Similar

S. Bai et al., “TouchStone: Evaluating Vision-Language Models by Language Models” arXiv.org, Aug. 31, 2023. https://arxiv.org/abs/2308.16890
Original
2023
Cited alongside, same era.
O. Burda-Lassen, “Machine Translation of Folktales: small-data-driven and LLM-based approaches,” ACL Anthology, Sep. 01, 2023. https://aclanthology.org/2023.clasp-1.8/
2023
Cited alongside, same era.
W. Dai et al., “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” arXiv.org, May 11, 2023. https://arxiv.org/abs/2305.06500
Original
2023
Cited alongside, same era.
C. Fu et al., “MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language models” arXiv.org, Jun. 23, 2023. https://arxiv.org/abs/2306.13394
Original
2023
Cited alongside, same era.

Then

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” arXiv.org, Apr. 17, 2023. https://arxiv.org/abs/2304.08485
Original
2023
Later among the works it cites.
S. Yin et al., “Woodpecker: Hallucination Correction for Multimodal large Language models,” arXiv.org, Oct. 24, 2023. https://arxiv.org/abs/2310.16045
Original
2023
Later among the works it cites.
B. Zheng et al., “Image captioning for cultural artworks: a case study on ceramics,” Multimedia Systems, vol. 29, no. 6, pp. 3223–3243, Sep. 2023, doi: 10.1007/s00530-023-01178-8
2023
Later among the works it cites.
Y. Cao, W. Li, J. Li, Y. Yuan, A. Karamolegkou, and D. Hershcovich, “Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing,” arXiv.org, Feb. 08, 2024. https://arxiv.org/abs/2402.06015
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

S. Huang et al., “Language Is Not All You Need: Aligning Perception with Language Models,” arXiv.org, Feb. 27, 2023

2023

Cited alongside, same era.

V. Krishna et al., “IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images,” arXiv.org, May 12, 2023. https://arxiv.org/abs/2305.10438

Original

2023

Cited alongside, same era.

C. Li et al., “Multimodal foundation models: from specialists to General-Purpose Assistants,” arXiv.org, Sep. 18, 2023. https://arxiv.org/abs/2309.10020

Original

2023

Cited alongside, same era.

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large Vision-Language models,” arXiv.org, May 17, 2023. https://arxiv.org/abs/2305.10355

Original

2023

Cited alongside, same era.

J. Kharchenko, T. Roosta, A. Chadha, and C. Shah, “How well do LLMs represent values across cultures? Empirical analysis of LLM responses based on Hofstede Cultural dimensions,” arXiv.org, Jun. 21, 2024. https://arxiv.org/abs/2406.14805https://arxiv.org/abs/2302.14045

Original

2024

Closest in time.

B. McKinzie et al., “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training,” arXiv.org, Mar. 14, 2024. https://arxiv.org/abs/2403.09611

Original

2024

Closest in time.

S. Nayak et al., “Benchmarking Vision language models for cultural understanding,” arXiv.org, Jul. 15, 2024. https://arxiv.org/abs/2407.10920

Original

2024

Closest in time.

S. M. T. I. Tonmoy et al., “A comprehensive survey of hallucination mitigation techniques in large language models,” arXiv.org, Jan. 02, 2024. https://arxiv.org/abs/2401.01313

Original

2024

Closest in time.