Fetching the paper…
Reading the bibliography…
In this paper, we study the problem of uncertainty estimation and calibration for LLMs.
Language models are few-shot learners
Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Benchmarking bayesian deep learning with diabetic retinopathy diagnosis
Filos, Angelos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, Yarin Gal. 2019 · 1912
Earlier work this paper cites.
Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods
Platt, John, et al. 1999 · 1999
Earlier work this paper cites.
Random forests
Breiman, Leo. 2001 · 2001
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Papineni, Kishore, Salim Roukos, Todd Ward, Wei jing Zhu. 2002 · 2002
Earlier work this paper cites.
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, Bianca, Charles Elkan. 2002 · 2002
Earlier work this paper cites.
Calibration of pre-trained transformers
Desai, Shrey, Greg Durrett. 2020 · 2003
Earlier work this paper cites.
ORANGE: a method for evaluating automatic evaluation metrics for machine translation
Lin, Chin-Yew, Franz Josef Och. 2004b · 2004
Earlier work this paper cites.
Measuring massive multitask language understanding
Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt. 2020 · 2009
Earlier work this paper cites.
Scikit-learn: Machine learning in python
Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011 · 2011
Earlier work this paper cites.
Statistical Decision Theory and Bayesian Analysis
Berger, J.O. 2013 · 2013
Earlier work this paper cites.
Findings of the 2014 workshop on statistical machine translation
Bojar, Ondřej, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014 · 2014
Earlier work this paper cites.
How the machine ‘thinks’: Understanding opacity in machine learning algorithms
Burrell, J. 2016 · 2016
Earlier work this paper cites.
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Yarin, Zoubin Ghahramani. 2016 · 2016
Earlier work this paper cites.
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, Andre, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, Sebastian Thrun. 2017 · 2017
Earlier work this paper cites.
On calibration of modern neural networks
Guo, Chuan, Geoff Pleiss, Yu Sun, Kilian Q Weinberger. 2017 · 2017
Earlier work this paper cites.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
Joshi, Mandar, Eunsol Choi, Daniel S Weld, Luke Zettlemoyer. 2017 · 2017
Earlier work this paper cites.
Simple and scalable predictive uncertainty estimation using deep ensembles
Lakshminarayanan, Balaji, Alexander Pritzel, Charles Blundell. 2017 · 2017
Earlier work this paper cites.
Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling
Ramos, Sebastian, Stefan Gehrig, Peter Pinggera, Uwe Franke, Carsten Rother. 2017 · 2017
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019
Earlier work this paper cites.
Unsupervised quality estimation for neural machine translation
Fomicheva, Marina, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia. 2020 · 2020
Earlier work this paper cites.
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, Moloud, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. 2021 · 2021
Cited alongside, same era.
Uncertainty estimation in autoregressive structured prediction
Malinin, Andrey, Mark Gales. 2021 · 2021
Cited alongside, same era.
Revisiting the calibration of modern neural networks
Minderer, Matthias, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, Mario Lucic. 2021 · 2021
Cited alongside, same era.
Can explanations be useful for calibrating black box models?
Ye, Xi, Greg Durrett. 2021 · 2021
Cited alongside, same era.
Knowing more about questions can help: Improving calibration in question answering
Zhang, Shujian, Chengyue Gong, Eunsol Choi. 2021 · 2021
Generating with confidence: Uncertainty quantification for black-box large language models
Lin, Zhen, Shubhendu Trivedi, Jimeng Sun. 2023 · 2023
Later among the works it cites.
Liu, Kevin, Stephen Casper, Dylan Hadfield-Menell, Jacob Andreas. 2023 · 2023
Later among the works it cites.
Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models
Manakul, Potsawee, Adian Liusie, Mark JF Gales. 2023 · 2023
Later among the works it cites.
Quach, Victor, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, Regina Barzilay. 2023 · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Discovering latent knowledge in language models without supervision
Burns, Collin, Haotian Ye, Dan Klein, Jacob Steinhardt. 2022 · 2022
Cited alongside, same era.
Language models (mostly) know what they know
Kadavath, Saurav, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022 · 2022
Cited alongside, same era.
Towards collaborative neural-symbolic graph semantic parsing via uncertainty
Lin, Zi, Jeremiah Zhe Liu, Jingbo Shang. 2022 · 2022
Cited alongside, same era.
Reducing conversational agents’ overconfidence through linguistic calibration
Mielke, Sabrina J, Arthur Szlam, Emily Dinan, Y-Lan Boureau. 2022 · 2022
Cited alongside, same era.
Training language models to follow instructions with human feedback
Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022
Cited alongside, same era.
Re-examining calibration: The case of question answering
Si, Chenglei, Chen Zhao, Sewon Min, Jordan Boyd-Graber. 2022 · 2022
Cited alongside, same era.
Uncertainty quantification with pre-trained language models: A large-scale empirical analysis
Xiao, Yuxin, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency. 2022 · 2022
Cited alongside, same era.
Rawte, Vipula, Amit Sheth, Amitava Das. 2023 · 2023
Later among the works it cites.
The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models
Slobodkin, Aviv, Omer Goldman, Avi Caciularu, Ido Dagan, Shauli Ravfogel. 2023 · 2023
Later among the works it cites.
Tian, Katherine, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D Manning. 2023 · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023
Later among the works it cites.
Reducing llm hallucinations using epistemic neural networks
Verma, Shreyas, Kien Tran, Yusuf Ali, Guangyu Min. 2023 · 2023
Later among the works it cites.
A study on the calibration of in-context learning
Zhang, Hanlin, Yi-Fan Zhang, Yaodong Yu, Dhruv Madeka, Dean Foster, Eric Xing, Hima Lakkaraju, Sham Kakade. 2023 · 2023
Later among the works it cites.
Batch calibration: Rethinking calibration for in-context learning and prompt engineering
Zhou, Han, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, Subhrajit Roy. 2023 · 2023
Later among the works it cites.
Distinguishing the knowable from the unknowable with language models
Ahdritz, Gustaf, Tian Qin, Nikhil Vyas, Boaz Barak, Benjamin L Edelman. 2024 · 2024
Closest in time.
Llama 3 model card URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
AI@Meta. 2024 · 2024
Closest in time.
Inside: Llms’ internal states retain the power of hallucination detection
Chen, Chao, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, Jieping Ye. 2024 · 2024
Closest in time.
Do llms know about hallucination? an empirical investigation of llm’s hidden states
Duan, Hanyu, Yi Yang, Kar Yan Tam. 2024 · 2024
Closest in time.
Gemma doi: 10.34740/KAGGLE/M/3301
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, et al. 2024 · 2024
Closest in time.
Inference-time intervention: Eliciting truthful answers from a language model
Li, Kenneth, Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg. 2024 · 2024
Closest in time.
Language models with conformal factuality guarantees
Mohri, Christopher, Tatsunori Hashimoto. 2024 · 2024
Closest in time.
Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a
Plaut, Benjamin, Khanh Nguyen, Tu Trinh. 2024 · 2024
Closest in time.
Unsupervised real-time hallucination detection based on the internal states of large language models
Su, Weihang, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, Yiqun Liu. 2024 · 2024
Closest in time.
Hallucination is inevitable: An innate limitation of large language models
Xu, Ziwei, Sanjay Jain, Mohan Kankanhalli. 2024 · 2024
Closest in time.