Bioformer: an efficient transformer language model for biomedical text mining

Understand

Pretrained language models such as Bidirectional Encoder Representations from Transformers (BERT) have achieved state-of-the-art performance in natural language processing (NLP) tasks.

• Recently, BERT has been adapted to the biomedical domain.
• Despite the effectiveness, these models have hundreds of millions of parameters and are computationally expensive when applied to large-scale NLP applications.
• We hypothesized that the number of parameters of the original BERT can be dramatically reduced with minor impact on performance.