Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency

Understand

Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches, with two major types being particularly prominent: information-based, which focus on model confidence expressed as token probabilities, and consistency-based, which assess the semantic relationship between multiple outputs generated using repeated sampling.

• Several recent methods have combined these two approaches to boost UQ performance.
• However, they sometimes fail to outperform much simpler baseline methods.
• Our work discusses the fundamental approach to constructing uncertainty measures that directly links uncertainty with the minimum Bayes risks achieved by LLM decoding.