Large Language Model Confidence Estimation via Black-Box Access

Understand

Estimating uncertainty or confidence in the responses of a model can be significant in evaluating trust not only in the responses, but also in the model as a whole.

• In this paper, we explore the problem of estimating confidence for responses of large language models (LLMs) with simply black-box or query access to them.
• We propose a simple and extensible framework where, we engineer novel features and train a (interpretable) model (viz.
• logistic regression) on these features to estimate the confidence.