Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems

Understand

Although large language models (LLMs) are becoming increasingly capable of solving challenging real-world tasks, accurately quantifying their uncertainty remains a critical open problem--one that limits their applicability in high-stakes domains.

• This challenge is further compounded by the closed-source, black-box nature of many state-of-the-art LLMs.
• Moreover, LLM-based systems can be highly sensitive to the prompts that bind them together, which often require significant manual tuning (i.e., prompt engineering).
• In this work, we address these challenges by viewing LLM-based systems through a Bayesian lens.