Methods to Estimate Large Language Model Confidence

Understand

Large Language Models have difficulty communicating uncertainty, which is a significant obstacle to applying LLMs to complex medical tasks.

• This study evaluates methods to measure LLM confidence when suggesting a diagnosis for challenging clinical vignettes.
• GPT4 was asked a series of challenging case questions using Chain of Thought and Self Consistency prompting.
• Multiple methods were investigated to assess model confidence and evaluated on their ability to predict the models observed accuracy.