Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings

Understand

Models based on the transformer architecture, such as BERT, have marked a crucial step forward in the field of Natural Language Processing.

• Importantly, they allow the creation of word embeddings that capture important semantic information about words in context.
• However, as single entities, these embeddings are difficult to interpret and the models used to create them have been described as opaque.
• Binder and colleagues proposed an intuitive embedding space where each dimension is based on one of 65 core semantic features.