2019

TENER: Adapting Transformer Encoder for Named Entity Recognition

Yan, Hang, Deng, Bocao, Li, Xiaonan et al.

Understand

The Bidirectional long short-term memory networks (BiLSTM) have been widely used as an encoder in models solving the named entity recognition (NER) task.

Recently, the Transformer is broadly adopted in various Natural Language Processing (NLP) tasks owing to its parallelism and advantageous performance.
Nevertheless, the performance of the Transformer in NER is not as good as it is in other NLP tasks.
In this paper, we propose TENER, a NER architecture adopting adapted Transformer Encoder to model the character-level features and word-level features.

Built on

Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001 · 2001
Earlier work this paper cites.
Introduction to the conll-2003 shared task: Language-independent named entity recognition
Erik F. Tjong Kim Sang and Fien De Meulder. 2003 · 2003
Earlier work this paper cites.
The third international chinese language processing bakeoff: Word segmentation and named entity recognition
Gina-Anne Levow. 2006 · 2006
Earlier work this paper cites.
Natural language processing (almost) from scratch
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011 · 2011
Earlier work this paper cites.
Ontonotes release 4.0 LDC2011T03
Ralph Weischedel. 2011 · 2011
Earlier work this paper cites.
Named entity recognition with bilingual constraints
Wanxiang Che, Mengqiu Wang, Christopher D. Manning, and Ting Liu. 2013 · 2013
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 · 2014
Earlier work this paper cites.
Bidirectional lstm-crf models for sequence tagging
Original
Zhiheng Huang, Wei Xu, and Kai Yu. 2015 · 2015
Earlier work this paper cites.
Named entity recognition for chinese social media with jointly trained embeddings
Nanyun Peng and Mark Dredze. 2015 · 2015
Earlier work this paper cites.
Named entity recognition with bidirectional lstm-cnns
Jason P. C. Chiu and Eric Nichols. 2016 · 2016
Earlier work this paper cites.
Character-based LSTM-CRF with radical-level features for chinese named entity recognition
Chuanhai Dong, Jiajun Zhang, Chengqing Zong, Masanori Hattori, and Hui Di. 2016 · 2016
Earlier work this paper cites.
Neural architectures for named entity recognition
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016 · 2016
Earlier work this paper cites.

Similar

End-to-end sequence labeling via bi-directional lstm-cnns-crf
Xuezhe Ma and Eduard Hovy. 2016 · 2016
Cited alongside, same era.
End-to-end relation extraction using lstms on sequences and tree structures
Original
Makoto Miwa and Mohit Bansal. 2016 · 2016
Cited alongside, same era.
Applying named entity recognition and co-reference resolution for segmenting english texts
Pavlina Fragkou. 2017 · 2017
Cited alongside, same era.
Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017 · 2017
Cited alongside, same era.
Cyclical learning rates for training neural networks
Leslie N. Smith. 2017 · 2017
Cited alongside, same era.
Fast and accurate entity recognition with iterated dilated convolutions
Original
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Neural question generation from text: A preliminary study
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017 · 2017
Cited alongside, same era.
Contextual string embeddings for sequence labeling
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018 · 2018
Cited alongside, same era.
Sequence labeling: A practical approach
Original
Adnan Akhundov, Dietrich Trautmann, and Georg Groh. 2018 · 2018
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Robust lexical features for improved neural network named-entity recognition
Original
Abbas Ghaddar and Philippe Langlais. 2018 · 2018
Cited alongside, same era.

Then

Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Later among the works it cites.
Self-attention with relative position representations
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018 · 2018
Later among the works it cites.
Learning better internal structure of words for sequence labeling
Yingwei Xin, Ethan Hart, Vibhuti Mahajan, and Jean-David Ruvini. 2018 · 2018
Later among the works it cites.
Five-stroke based cnn-birnn-crf network for chinese named entity recognition
Fan Yang, Jianhu Zhang, Gongshen Liu, Jie Zhou, Cheng Zhou, and Huanrong Sun. 2018 · 2018
Later among the works it cites.
Hybrid semi-markov CRF for neural sequence labeling
Original
Zhi-Xiu Ye and Zhen-Hua Ling. 2018 · 2018
Later among the works it cites.
Sentence-state LSTM for text representation
Yue Zhang, Qi Liu, and Linfeng Song. 2018 · 2018
Later among the works it cites.
Chinese NER using lattice LSTM
Yue Zhang and Jie Yang. 2018 · 2018
Later among the works it cites.
GRN: Gated relation network to enhance convolutional neural network for named entity recognition
Hui Chen, Zijia Lin, Guiguang Ding, Jianguang Lou, Yusen Zhang, and Borje Karlsson. 2019 · 2019
Closest in time.
Transformer-xl: Attentive language models beyond a fixed-length context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019 · 2019
Closest in time.
Star-transformer
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019 · 2019
Closest in time.
Music transformer: Generating music with long-term structure
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019 · 2019
Closest in time.
Contextualized non-local neural networks for sequence learning
Pengfei Liu, Shuaichen Chang, Xuanjing Huang, Jian Tang, and Jackie Chi Kit Cheung. 2019 · 2019
Closest in time.
CAN-NER: convolutional attention network for chinese named entity recognition
Yuying Zhu and Guoxin Wang. 2019 · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…