Legal Transformer Models May Not Always Help

To pretrain or not to pretrain: Examining the benefits of pretraining on resource rich tasks

Original

Wang, S., Khabsa, M., and Ma, H. (2020) · 2006

Earlier work this paper cites.

Lexa: Towards automatic legal citation classification

Galgani, F. and Hoffmann, A. (2010) · 2010

Earlier work this paper cites.

UCI machine learning repository

Dua, D. and Graff, C. (2017) · 2017

Earlier work this paper cites.

Vse++: Improving visual-semantic embeddings with hard negatives

Faghri, F., Fleet, D. J., Kiros, J. R., and Fidler, S. (2018) · 2018

Earlier work this paper cites.

Large-scale multi-label text classification on EU legislation

Chalkidis, I., Fergadiotis, E., Malakasiotis, P., and Androutsopoulos, I. (2019) · 2019

Earlier work this paper cites.

Law and word order: Nlp in legal tech

DALE, R. (2019) · 2019

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019) · 2019

Cited alongside, same era.

Parameter-efficient transfer learning for nlp

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019) · 2019

Cited alongside, same era.

Biobert: a pre-trained biomedical language representation model for biomedical text mining

Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. (2019) · 2019

Cited alongside, same era.

Roberta: A robustly optimized bert pretraining approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019) · 2019

Cited alongside, same era.

Legal-bert: The muppets straight out of law school

Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020) · 2020

Later among the works it cites.

Don’t stop pretraining: Adapt language models to domains and tasks

Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. (2020) · 2020

Later among the works it cites.

On the importance of pre-training data volume for compact language models

Micheli, V., d’Hoffschmidt, M., and Fleuret, F. (2020) · 2020

Later among the works it cites.

Neural contract element extraction revisited: Letters from sesame street

Chalkidis, I., Fergadiotis, M., Malakasiotis, P., and Androutsopoulos, I. (2021) · 2021

Closest in time.

When does pretraining help? assessing self-supervised learning for law and the casehold dataset

Original

Zheng, L., Guha, N., Anderson, B. R., Henderson, P., and Ho, D. E. (2021) · 2021

Closest in time.