Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences

Understand

Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks.

• The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks.
• One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism.
• To overcome this, long sequence transformer models, e.g.