Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models

Understand

A variety of contextualised language models have been proposed in the NLP community, which are trained on diverse corpora to produce numerous Neural Language Models (NLMs).

• However, different NLMs have reported different levels of performances in downstream NLP applications when used as text representations.
• We propose a sentence-level meta-embedding learning method that takes independently trained contextualised word embedding models and learns a sentence embedding that preserves the complementary strengths of the input source NLMs.
• Our proposed method is unsupervised and is not tied to a particular downstream task, which makes the learnt meta-embeddings in principle applicable to different tasks that require sentence representations.