In-context Learning with Retrieved Demonstrations for Language Models: A Survey

Understand

Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context.

• However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations.
• Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query.
• The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems.