LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity

Understand

Combining large language models during training or at inference time has shown substantial performance gain over component LLMs.

• This paper presents LLM-TOPLA, a diversity-optimized LLM ensemble method with three unique properties: (i) We introduce the focal diversity metric to capture the diversity-performance correlation among component LLMs of an ensemble.
• (ii) We develop a diversity-optimized ensemble pruning algorithm to select the top-k sub-ensembles from a pool of $N$ base LLMs.
• Our pruning method recommends top-performing LLM subensembles of size $S$, often much smaller than $N$.