DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration

Understand

Pretraining language models is still a challenge for many researchers due to its substantial computational costs.

• As such, there is growing interest in developing more affordable pretraining methods.
• One notable advancement in this area is the Cramming technique (Geiping and Goldstein, 2022), which enables the pretraining of BERT-style language models using just one GPU in a single day.
• Building on this innovative approach, we introduce the Dependency Agreement Cramming (DA-Cramming), an efficient framework that integrates information about dependency agreements into the pretraining process.