Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty

Uncertainties have a meaning: Information entropy as a quality measure for 3-d geological models

J Florian Wellmann and Klaus Regenauer-Lieb. 2012 · 2012

Earlier work this paper cites.

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, and 1 others. 2021 · 2021

Earlier work this paper cites.

Uncertainty estimation in autoregressive structured prediction

Andrey Malinin and Mark Gales. 2021 · 2021

Earlier work this paper cites.

Measuring progress on scalable oversight for large language models

Original

Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, and 27 others. 2022 · 2022

Earlier work this paper cites.

Language models (mostly) know what they know

Original

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022 · 2022

Earlier work this paper cites.

TruthfulQA: Measuring how models mimic human falsehoods

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022 · 2022

Earlier work this paper cites.

Demystifying prompts in language models via perplexity estimation

Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023 · 2023

Earlier work this paper cites.

Won’t get fooled again: Answering questions with false premises

Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023 · 2023

Earlier work this paper cites.

Survey of hallucination in natural language generation

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023 · 2023

Earlier work this paper cites.

Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023 · 2023

Earlier work this paper cites.

Self-refine: Iterative refinement with self-feedback

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023 · 2023

Earlier work this paper cites.

When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 · 2023

Earlier work this paper cites.

SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models

Potsawee Manakul, Adian Liusie, and Mark Gales. 2023 · 2023

Earlier work this paper cites.

Reflexion: language agents with verbal reinforcement learning

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023 · 2023

Earlier work this paper cites.

Recitation-augmented language models

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023 · 2023

Earlier work this paper cites.

Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023 · 2023

Cited alongside, same era.

Freshllms: Refreshing large language models with search engine augmentation

Original

Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. 2023 · 2023

Cited alongside, same era.

Self-consistency improves chain of thought reasoning in language models

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 · 2023

Cited alongside, same era.

Examining inter-consistency of large language models collaboration: An in-depth analysis via debate

Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. 2023 · 2023

Cited alongside, same era.

SAC 3 : Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency

Halo: An ontology for representing and categorizing hallucinations in large language models

Original

Navapat Nananukul and Mayank Kejriwal. 2024 · 2024

Closest in time.

Kernel language entropy: Fine-grained uncertainty quantification for LLMs from semantic similarities

Alexander V Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024 · 2024

Closest in time.

Gpt-4 technical report

Original

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024 · 2024

Closest in time.

Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space

Xin Qiu and Risto Miikkulainen. 2024 · 2024

Closest in time.