Automated Concatenation of Embeddings for Structured Prediction

Understand

Pretrained contextualized embeddings are powerful word representations for structured prediction tasks.

• Recent work found that better word representations can be obtained by concatenating different types of embeddings.
• However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem.
• In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search.