Improving In-Context Learning with Small Language Model Ensembles

Understand

Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited.

• While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources.
• In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods.
• We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs).