Ambiguity-Aware In-Context Learning with Large Language Models

Understand

In-context learning (ICL) i.e.

• showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required.
• However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL.
• One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task.