Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling

Understand

Ensembling BERT models often significantly improves accuracy, but at the cost of significantly more computation and memory footprint.

• In this work, we propose Multi-CLS BERT, a novel ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model.
• Multi-CLS BERT uses multiple CLS tokens with a parameterization and objective that encourages their diversity.
• Thus instead of fine-tuning each BERT model in an ensemble (and running them all at test time), we need only fine-tune our single Multi-CLS BERT model (and run the one model at test time, ensembling just the multiple final CLS embeddings).