Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memories

Understand

Pre-trained language models (PLMs) demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain.

• Although continued pre-training on a large domain-specific corpus is effective, it is costly to tune all the parameters on the domain.
• In this paper, we investigate whether we can adapt PLMs both effectively and efficiently by only tuning a few parameters.
• Specifically, we decouple the feed-forward networks (FFNs) of the Transformer architecture into two parts: the original pre-trained FFNs to maintain the old-domain knowledge and our novel domain-specific adapters to inject domain-specific knowledge in parallel.