Similarity Analysis of Contextual Word Representation Models

Understand

This paper investigates contextual word representation models from the lens of similarity analysis.

• Given a collection of trained models, we measure the similarity of their internal representations and attention.
• Critically, these models come from vastly different architectures.
• We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation.