Conformal Information Pursuit for Interactively Guiding Large Language Models

Understand

A significant use case of instruction-finetuned Large Language Models (LLMs) is to solve question-answering tasks interactively.

• In this setting, an LLM agent is tasked with making a prediction by sequentially querying relevant information from the user, as opposed to a single-turn conversation.
• This paper explores sequential querying strategies that aim to minimize the expected number of queries.
• One such strategy is Information Pursuit (IP), a greedy algorithm that at each iteration selects the query that maximizes information gain or equivalently minimizes uncertainty.