Fetching the paper…
Reading the bibliography…
We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large.
An introduction to the bootstrap
Robert J Tibshirani and Bradley Efron · 1993
Earlier work this paper cites.
Good-turing frequency estimation without tears
William A Gale and Geoffrey Sampson · 1995
Earlier work this paper cites.
WordNet: An electronic lexical database
Christiane Fellbaum · 1998
Earlier work this paper cites.
On the convergence rate of good-turing estimators
David A McAllester and Robert E Schapire · 2000
Earlier work this paper cites.
Prediction, learning, and games
Nicolo Cesa-Bianchi and Gábor Lugosi · 2006
Earlier work this paper cites.
Distribution-dependent performance of the good-turing estimator for the missing mass
Mesrob I Ohannessian and Munther A Dahleh · 2010
Earlier work this paper cites.
The missing mass problem
Daniel Berend and Aryeh Kontorovich · 2012
Earlier work this paper cites.
Bayesian learning for neural networks
R. M. Neal · 2012
Earlier work this paper cites.
On the concentration of the missing mass
Daniel Berend and Aryeh Kontorovich · 2013
Earlier work this paper cites.
Zipf’s word frequency law in natural language: A critical review and future directions
Steven T Piantadosi · 2014
Earlier work this paper cites.
Weight uncertainty in neural network
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra · 2015
Earlier work this paper cites.
Uncertainty in deep learning
Yarin Gal · 2016
Earlier work this paper cites.
Deep exploration via bootstrapped dqn
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy · 2016
Earlier work this paper cites.
The expected missing mass under an entropy constraint
Daniel Berend, Aryeh Kontorovich, and Gil Zagdanski · 2017
Earlier work this paper cites.
TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer · 2017
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Depth uncertainty in neural networks
Javier Antorán, James Allingham, and José Miguel Hernández-Lobato · 2020
Earlier work this paper cites.
Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly
Nora Kassner and Hinrich Schütze · 2020
Earlier work this paper cites.
Uncertainty estimation in autoregressive structured prediction
Andrey Malinin and Mark Gales · 2020
Earlier work this paper cites.
AmbigQA: Answering ambiguous open-domain questions
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer · 2020
Earlier work this paper cites.
Entity-based knowledge conflicts in question answering
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh · 2021
Cited alongside, same era.
Calibrate before use: Improving few-shot performance of language models
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh · 2021
Cited alongside, same era.
Language models (mostly) know what they know
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, and et al · 2022
Cited alongside, same era.
Reducing conversational agents’ overconfidence through linguistic calibration
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau · 2022
Cited alongside, same era.
Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend · 2022
SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales · 2023
Later among the works it cites.
Epistemic neural networks
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy · 2023
Later among the works it cites.
Conformal nucleus sampling
Shauli Ravfogel, Yoav Goldberg, and Jacob Goldberger · 2023
Later among the works it cites.
Sac3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency
Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar · 2023
Later among the works it cites.
Distinguishing the knowable from the unknowable with language models, 2024
Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, and Benjamin L. Edelman · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Selective classification via neural network training dynamics
Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, and Nicolas Papernot · 2022
Cited alongside, same era.
Self-consistency improves chain of thought reasoning in language models
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou · 2022
Cited alongside, same era.
From predictions to decisions: The importance of joint predictive distributions
Zheng Wen, Ian Osband, Chao Qin, Xiuyuan Lu, Morteza Ibrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy · 2022
Cited alongside, same era.
Conformal prediction: A gentle introduction
Anastasios N Angelopoulos, Stephen Bates, et al · 2023
Cited alongside, same era.
The internal state of an LLM knows when its lying
Amos Azaria and Tom Mitchell · 2023
Cited alongside, same era.
Sparks of artificial general intelligence: Early experiments with gpt-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang · 2023
Cited alongside, same era.
Discovering latent knowledge in language models without supervision
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt · 2023
Cited alongside, same era.
Detecting hallucinations in large language models using semantic entropy
S. Farquhar, J. Kossen, L. Kuhn, and Yarin Gal · 2024
Closest in time.
Gemini: A family of highly capable multimodal models
Gemini Team, Google · 2024
Closest in time.
Decomposing uncertainty for large language models through input clarification ensembling
Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang · 2024
Closest in time.
Calibrating language models via augmented prompt ensembles
Mingjian Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Grosse, and Jimmy Ba · 2024
Closest in time.
Experts don’t cheat: Learning what you don’t know by predicting pairs
Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison · 2024
Closest in time.
Understanding the effects of iterative prompting on truthfulness
Satyapriya Krishna, Chirag Agarwal, and Himabindu Lakkaraju · 2024
Closest in time.
Are you sure? challenging llms leads to performance drops in the flipflop experiment, 2024
Philippe Laban, Lidiya Murakhovs’ka, Caiming Xiong, and Chien-Sheng Wu · 2024
Closest in time.
Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua · 2024
Closest in time.
Information theory: From coding to learning
Yury Polyanskiy and Yihong Wu · 2024
Closest in time.
On subjective uncertainty quantification and calibration in natural language generation, 2024
Ziyu Wang and Chris Holmes · 2024
Closest in time.
Mitigating llm hallucinations via conformal abstention
Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev · 2024
Closest in time.
Characterizing truthfulness in large language model generations with local intrinsic dimension
Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang · 2024
Closest in time.
Gal Yona, Roee Aharoni, and Mor Geva · 2024
Closest in time.
Knowing what LLMs do not know: A simple yet effective self-detection method
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin · 2024
Closest in time.