DagoBERT: Generating Derivational Morphology with a Pretrained Language Model

Understand

Can pretrained language models (PLMs) generate derivationally complex words? We present the first study investigating this question, taking BERT as the example PLM.

• We examine BERT's derivational capabilities in different settings, ranging from using the unmodified pretrained model to full finetuning.
• Our best model, DagoBERT (Derivationally and generatively optimized BERT), clearly outperforms the previous state of the art in derivation generation (DG).
• Furthermore, our experiments show that the input segmentation crucially impacts BERT's derivational knowledge, suggesting that the performance of PLMs could be further improved if a morphologically informed vocabulary of units were used.