Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach

R. E. Wright, “Logistic regression.” Reading and Understanding Multivariate Statistics , pp. 217–244, 1995

1995

Earlier work this paper cites.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

Original

2014

Earlier work this paper cites.

S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

Original

2016

Earlier work this paper cites.

D. R. S. Saputro and P. Widyaningsih, “Limited memory broyden-fletcher-goldfarb-shanno (l-bfgs) method for the parameter estimation on geographically weighted ordinal logistic regression model (gwolr),” AIP Conference Proceedings , 2017

2017

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

Z. Zhao, S. B. Cohen, and B. Webber, “Reducing quantity hallucinations in abstractive summarization,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 2237–2249. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.203

2020

Earlier work this paper cites.

A. Malinin and M. Gales, “Uncertainty estimation in autoregressive structured prediction,” arXiv preprint arXiv:2002.07650 , 2020

Original

2020

Earlier work this paper cites.

M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880. [Online]. Available: https://aclanthology.org/2020.acl-main.703

2020

Earlier work this paper cites.

I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150 , 2020

Original

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” in Findings of the Association for Computational Linguistics: EMNLP 2021 , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 3784–3803. [Online]. Available: https://aclanthology.org/2021.findings-emnlp.320

2021

Earlier work this paper cites.

Y. Xiao and W. Y. Wang, “On hallucination and predictive uncertainty in conditional language generation,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , P. Merlo, J. Tiedemann, and R. Tsarfaty, Eds. Online: Association for Computational Linguistics, Apr. 2021, pp. 2734–2744. [Online]. Available: https://aclanthology.org/2021.eacl-main.236

2021

Earlier work this paper cites.

W. Yuan, G. Neubig, and P. Liu, “Bartscore: Evaluating generated text as text generation,” Advances in Neural Information Processing Systems , vol. 34, pp. 27 263–27 277, 2021

2021

Earlier work this paper cites.

B. Wang and A. Komatsuzaki, “Gpt-j-6b: A 6 billion parameter autoregressive language model,” 2021

2021

Cited alongside, same era.

S. Das, S. Saha, and R. Srihari, “Diving deep into modes of fact hallucinations in dialogue systems,” in Findings of the Association for Computational Linguistics: EMNLP 2022 , Y. Goldberg, Z. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 684–699. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.48

2022

Cited alongside, same era.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

Original

2022

Cited alongside, same era.

W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223 , 2023

Original

2023

J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen, “HaluEval: A large-scale hallucination evaluation benchmark for large language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 6449–6464. [Online]. Available: https://aclanthology.org/2023.emnlp-main.397

2023

Later among the works it cites.

M. Lee, “A mathematical investigation of hallucination and creativity in gpt models,” Mathematics , vol. 11, no. 10, p. 2320, 2023

2023

Later among the works it cites.

A. Azaria and T. Mitchell, “The internal state of an LLM knows when it’s lying,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 967–976. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.68

2023

Later among the works it cites.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023