A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Video-based facial expression hallucination: A two-level hierarchical fusion approach

Jian Zhang, Yueting Zhuang, and Fei Wu. 2006 · 2006

Earlier work this paper cites.

Content matters: A qualitative analysis of verbal hallucinations

Nienke Moernaut, Isabella Leudar, and Thomas Verdooren. 2018 · 2018

Earlier work this paper cites.

Towards automatic learning of procedures from web instructional videos

Luowei Zhou, Chenliang Xu, and Jason Corso. 2018 · 2018

Earlier work this paper cites.

Parallelized convolutional recurrent neural network with spectral features for speech emotion recognition

Pengxu Jiang, Hongliang Fu, Huawei Tao, Peizhi Lei, and Li Zhao. 2019 · 2019

Earlier work this paper cites.

Streamlined dense video captioning

Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bohyung Han. 2019 · 2019

Earlier work this paper cites.

Multi-modal dense video captioning

Vladimir Iashin and Esa Rahtu. 2020 · 2020

Earlier work this paper cites.

An efficient framework for dense video captioning

Maitreya Suin and AN Rajagopalan. 2020 · 2020

Earlier work this paper cites.

Automated audio captioning with weakly supervised pre-training and word selection methods

Qichen Han, Weiqiang Yuan, Dong Liu, Xiang Li, and Zhen Yang. 2021 · 2021

Earlier work this paper cites.

Internal video inpainting by implicit long-range propagation

Hao Ouyang, Tengfei Wang, and Qifeng Chen. 2021 · 2021

Earlier work this paper cites.

Improving the performance of automated audio captioning via integrating the acoustic and semantic information

Original

Zhongjie Ye, Helin Wang, Dongchao Yang, and Yuexian Zou. 2021 · 2021

Earlier work this paper cites.

Maxm: Towards multilingual visual question answering

Original

Soravit Changpinyo, Linting Xue, Idan Szpektor, Ashish V Thapliyal, Julien Amelot, Michal Yarom, Xi Chen, and Radu Soricut. 2022 · 2022

Earlier work this paper cites.

Plausible may not be faithful: Probing object hallucination in vision-language pre-training

Original

Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2022 · 2022

Earlier work this paper cites.

Rarr: Researching and revising what language models say, using language models

Original

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2022 · 2022

Earlier work this paper cites.

Multi-granularity aggregation transformer for joint video-audio-text representation learning

Mengge He, Wenjing Du, Zhiquan Wen, Qing Du, Yutong Xie, and Qi Wu. 2022 · 2022

Earlier work this paper cites.

Diffusion models for video prediction and infilling

Original

Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. 2022 · 2022

Earlier work this paper cites.

Valhalla: Visual hallucination for machine translation. in 2022 ieee

Y Li, R Panda, Y Kim, C Chen, R Feris, D Cox, and N Vasconcelos. 2022 · 2022

Earlier work this paper cites.

Emscore: Evaluating video captioning via coarse-grained and fine-grained embedding matching

Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. 2022 · 2022

Earlier work this paper cites.

Adversarial semantic hallucination for domain generalized semantic segmentation

Gabriel Tjio, Ping Liu, Joey Tianyi Zhou, and Rick Siow Mong Goh. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Learning video representations from large language models

Original

Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2022 · 2022

Earlier work this paper cites.

Creating trustworthy llms: Dealing with hallucinations in healthcare ai

Original

Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023 · 2023

Earlier work this paper cites.

The internal state of an llm knows when its lying

Original

Amos Azaria and Tom Mitchell. 2023 · 2023

Earlier work this paper cites.

Touchstone: Evaluating vision-language models by language models

Original

Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. 2023 · 2023

Earlier work this paper cites.

Audiolm: a language modeling approach to audio generation

Original

Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2023 · 2023

Earlier work this paper cites.

Evaluating hallucinations in chinese large language models

Original

Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, et al. 2023 · 2023

Earlier work this paper cites.

Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios

Original

I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023 · 2023

Earlier work this paper cites.

Clearvid: Curriculum learning for video description

Original

Cheng-Yu Chuang and Pooyan Fazli. 2023 · 2023

Earlier work this paper cites.

Dola: Decoding by contrasting layers improves factuality in large language models

Original

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023 · 2023

Earlier work this paper cites.

Chatlaw: Open-source legal large language model with integrated external knowledge bases

Original

Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023 · 2023

Earlier work this paper cites.

Chain-of-verification reduces hallucination in large language models

Original

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023 · 2023

Earlier work this paper cites.

Lp-musiccaps: Llm-based pseudo music captioning

Original

SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023 · 2023

Cited alongside, same era.

Halo: Estimation and reduction of hallucinations in open-source weak large language models

Original

Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. 2023 · 2023

Cited alongside, same era.

Text-to-audio generation using instruction-tuned llm and latent diffusion model

Original

Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. 2023 · 2023

Cited alongside, same era.

Compa: Addressing the gap in compositional reasoning in audio-language models

Original

Sreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi, Chandra Kiran Evuru, S Ramaneswaran, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2023 · 2023

Cited alongside, same era.

Explaining legal concepts with augmented large language models (gpt-4)

Original

Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. 2023 · 2023

Later among the works it cites.

Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

Original

Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2023 · 2023

Later among the works it cites.

Aligning large multimodal models with factually augmented rlhf

Original

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023 · 2023

Later among the works it cites.

A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation

Original

Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023 · 2023

Later among the works it cites.