BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?

Understand

Pretrained transformer models have achieved state-of-the-art results in many tasks and benchmarks recently.

• Many state-of-the-art Language Models (LMs), however, do not scale well above the threshold of 512 input tokens.
• In specialized domains though (such as legal, scientific or biomedical), models often need to process very long text (sometimes well above 10000 tokens).
• Even though many efficient transformers have been proposed (such as Longformer, BigBird or FNet), so far, only very few such efficient models are available for specialized domains.