A Cohesive Distillation Architecture for Neural Language Models

Understand

A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size, which prevents research groups without a necessary hardware infrastructure from participating in the development process.

• This study investigates methods for Knowledge Distillation (KD) to provide efficient alternatives to large-scale models.
• In this context, KD means extracting information about language encoded in a Neural Network and Lexical Knowledge Databases.
• We developed two methods to test our hypothesis that efficient architectures can gain knowledge from LMs and extract valuable information from lexical sources.