BERT Rediscovers the Classical NLP Pipeline

Multi-task deep neural networks for natural language understanding

Original

Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019 · 1901

Earlier work this paper cites.

Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals

Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009 · 2010

Earlier work this paper cites.

Distributed representations of words and phrases and their compositionality

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013 · 2013

Earlier work this paper cites.

OntoNotes release 5.0 LDC2013T19

Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013 · 2013

Earlier work this paper cites.

The Stanford CoreNLP natural language processing toolkit

Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014 · 2014

Earlier work this paper cites.

A gold standard dependency corpus for English

Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014 · 2014

Earlier work this paper cites.

Semantic proto-roles

Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015 · 2015

Earlier work this paper cites.

Does string-based neural MT learn source syntax?

Xing Shi, Inkit Padhi, and Kevin Knight. 2016 · 2016

Cited alongside, same era.

Semantic proto-role labeling

Adam Teichert, Adam Poliak, Benjamin Van Durme, and Matthew Gormley. 2017 · 2017

Cited alongside, same era.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Cited alongside, same era.

On internal language representations in deep learning: An analysis of machine translation and speech recognition

Yonatan Belinkov. 2018 · 2018

Cited alongside, same era.

Deep RNNs encode soft hierarchical syntax

Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018 · 2018

Cited alongside, same era.

What you can cram into a single $&​#* vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018 · 2018

Targeted syntactic evaluation of language models

Rebecca Marvin and Tal Linzen. 2018 · 2018

Later among the works it cites.

Collecting diverse natural language inference problems for sentence representation evaluation

Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018 · 2018

Later among the works it cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Later among the works it cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Closest in time.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Closest in time.