Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding

Divergence measures based on the shannon entropy

J. Lin. 1991 · 1991

Earlier work this paper cites.

Microsoft coco: Common objects in context

Original

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015 · 2015

Earlier work this paper cites.

Visual genome: Connecting language and vision using crowdsourced dense image annotations

Original

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. 2016 · 2016

Earlier work this paper cites.

Object hallucination in image captioning

Original

Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2019 · 2019

Earlier work this paper cites.

Scaling instruction-finetuned language models

Original

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022 · 2022

Earlier work this paper cites.

Shikra: Unleashing multimodal llm’s referential dialogue magic

Original

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023 · 2023

Earlier work this paper cites.

Instructblip: Towards general-purpose vision-language models with instruction tuning

Original

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023 · 2023

Earlier work this paper cites.

Detoxifying text with MaRCo: Controllable revision with experts and anti-experts

Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. 2023 · 2023

Earlier work this paper cites.

Faithscore: Evaluating hallucinations in large vision-language models

Original

Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023 · 2023

Earlier work this paper cites.

Critic-guided decoding for controlled text generation

Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2023 · 2023

Earlier work this paper cites.

Mitigating object hallucinations in large vision-language models through visual contrastive decoding

Original

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2023 · 2023

Earlier work this paper cites.

Evaluating object hallucination in large vision-language models

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023c · 2023

Cited alongside, same era.

Visual instruction tuning

Original

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 · 2023

Cited alongside, same era.

Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization

Original

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023 · 2023

Cited alongside, same era.

Minigpt-4: Enhancing vision-language understanding with advanced large language models

Original

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023 · 2023

Cited alongside, same era.

Multi-modal hallucination control by visual information grounding

Can LLMs recognize toxicity? a structured investigation framework and toxicity metric

Hyukhun Koh, Dohyung Kim, Minwoo Lee, and Kyomin Jung. 2024 · 2024

Closest in time.

Volcano: Mitigating multimodal hallucination through self-feedback guided revision

Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. 2024c · 2024

Closest in time.

Mitigating hallucinations in large vision-language models (LVLMs) via language-contrastive decoding (LCD)

Avshalom Manevich and Reut Tsarfaty. 2024 · 2024

Closest in time.

Hello gpt-4o

OpenAI. 2024 · 2024

Closest in time.

Contrastive region guidance: Improving grounding in vision-language models without training

Original

David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. 2024 · 2024

Closest in time.

Mitigating hallucinations in large vision-language models with instruction contrastive decoding