Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models

Introduction to the conll-2003 shared task: Language-independent named entity recognition

E. F. T. K. Sang and F. D. Meulder · 2003

Earlier work this paper cites.

Natural language processing (almost) from scratch

R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. P. Kuksa · 2011

Earlier work this paper cites.

Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks

D.-H. Lee · 2013

Earlier work this paper cites.

Glove: Global vectors for word representation

J. Pennington, R. Socher, and C. D. Manning · 2014

Earlier work this paper cites.

Named entity recognition with bidirectional lstm-cnns

J. P. C. Chiu and E. Nichols · 2015

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

G. E. Hinton, O. Vinyals, and J. Dean · 2015

Earlier work this paper cites.

Neural architectures for named entity recognition

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer · 2016

Cited alongside, same era.

End-to-end sequence labeling via bi-directional lstm-cnns-crf

Original

X. Ma and E. H. Hovy · 2016

Cited alongside, same era.

Name tagging for low-resource incident languages based on expectation-driven learning

B. Zhang, X. Pan, T. Wang, A. Vaswani, H. Ji, K. Knight, and D. Marcu · 2016

Cited alongside, same era.

Fast and accurate entity recognition with iterated dilated convolutions

E. Strubell, P. Verga, D. Belanger, and A. McCallum · 2017

Cited alongside, same era.

Attention is all you need

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin · 2017

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding

Deep contextualized word representations

Original

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. S. Zettlemoyer · 2018

Later among the works it cites.

Improving language understanding by generative pre-training

A. Radford · 2018

Later among the works it cites.

Distilling bert models with spacy

Y. Peirsman · 2019

Closest in time.

Smaller, faster, cheaper, lighter: Introducing distilbert, a distilled version of bert

V. Sanh · 2019

Closest in time.

Small and practical bert models for sequence labeling

H. Tsai, J. Riesa, M. Johnson, N. Arivazhagan, X. Li, and A. Archer · 2019

Closest in time.