2019

Span-based Joint Entity and Relation Extraction with Transformer Pre-training

Eberts, Markus, Ulges, Adrian

Understand

We introduce SpERT, an attention model for span-based joint entity and relation extraction.

Our key contribution is a light-weight reasoning on BERT embeddings, which features entity recognition and filtering, as well as relation classification with a localized, marker-free context representation.
The model is trained using strong within-sentence negative samples, which are efficiently extracted in a single BERT pass.
These aspects facilitate a search over all spans in the sentence.

Built on

Dan Roth and Wen-tau Yih, ‘A Linear Programming Formulation for Global Inference in Natural Language Tasks’, in Proc. of CoNLL 2004 at HLT-NAACL 2004
2004
Earlier work this paper cites.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo, ‘Development of a Benchmark Corpus to Support the Automatic Extraction of Drug-related Adverse Effects from Medical Case Reports’, J. of Biomedical Informatics
2012
Earlier work this paper cites.
Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng, ‘Semantic Compositionality Through Recursive Matrix-vector Spaces’, in Proc. of EMNLP-CoNLL 2012
2012
Earlier work this paper cites.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, ‘Neural Machine Translation by Jointly Learning to Align and Translate’, CoRR
Original
2014
Earlier work this paper cites.
Makoto Miwa and Yutaka Sasaki, ‘Modeling Joint Entity and Relation Extraction with Table Representation’, in Proc. of EMNLP 2014
2014
Earlier work this paper cites.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao, ‘Relation Classification via Convolutional Deep Neural Network’, in Proc. of COLING 2014
2014
Earlier work this paper cites.
Dongxu Zhang and Dong Wang, ‘Relation Classification via Recurrent Neural Network’, CoRR
Original
2015
Earlier work this paper cites.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy, ‘Table Filling Multi-Task Recurrent Neural Network for Joint Entity and Relation Extraction’, in Proc. of COLING 2016
2016
Earlier work this paper cites.
Fei Li, Yue Zhang, Meishan Zhang, and Donghong Ji, ‘Joint Models for Extracting Adverse Drug Events from Biomedical Text’, in Proc. of IJCAI 2016
2016
Earlier work this paper cites.
Makoto Miwa and Mohit Bansal, ‘End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures’, in Proc. of ACL 2016
2016
Earlier work this paper cites.
Rico Sennrich, Barry Haddow, and Alexandra Birch, ‘Neural Machine Translation of Rare Words with Subword Units’, in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
2016
Earlier work this paper cites.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer, ‘End-to-end Neural Coreference Resolution’, in Proc. of EMNLP 2017
2017
Earlier work this paper cites.
Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji, ‘A neural joint model for entity and relation extraction from biomedical text’, BMC Bioinformatics
2017
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, ‘Attention is All you Need’, in Advances in Neural Information Processing Systems 30
2017
Earlier work this paper cites.

Similar

Meishan Zhang, Yue Zhang, and Guohong Fu, ‘End-to-End Neural Relation Extraction with Global Optimization’, in Proc. of EMNLP 2017
2017
Cited alongside, same era.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu, ‘Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme’, in Proc. of ACL 2017
2017
Cited alongside, same era.
Peng Zhou, Suncong Zheng, Jiaming Xu, Zhenyu Qi, Hongyun Bao, and Bo Xu, ‘Joint Extraction of Multiple Relations and Entities by Using a Hybrid Neural Network’, in Proc. of CCL 2017
2017
Cited alongside, same era.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder, ‘Adversarial training for multi-context joint entity and relation extraction’, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
2018
Cited alongside, same era.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder, ‘Joint entity recognition and relation extraction as a multi-head selection problem’, Expert Systems with Applications
2018
Cited alongside, same era.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer, ‘Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling’, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
2018
Cited alongside, same era.
Kenton Lee, Luheng He, and Luke Zettlemoyer, ‘Higher-Order Coreference Resolution with Coarse-to-Fine Inference’, in Proc. of NAACL-HLT 2018
2018
Cited alongside, same era.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi, ‘Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction’, in Proc. of EMNLP 2018
2018
Cited alongside, same era.
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto, ‘A Span Selection Model for Semantic Role Labeling’, in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
2018
Cited alongside, same era.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, ‘Improving Language Understanding by Generative Pre-Training’, (2018)
2018
Cited alongside, same era.
Patrick Verga, Emma Strubell, and Andrew Mccallum, ‘Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction’, in Proc. of ACL-HLT 2018
2018
Cited alongside, same era.
Vikas Yadav and Steven Bethard, ‘A Survey on Recent Advances in Named Entity Recognition from Deep Learning models’, in Proc. of the 27th International Conference on Computational Linguistics
2018
Cited alongside, same era.

Then

Renjun Chi, Bin Wu, Linmei Hu, and Yunlei Zhang, ‘Enhancing Joint Entity and Relation Extraction with Language Modeling and Hierarchical Attention’, in Proc. APWeb-WAIM, LNCS 11641
2019
Closest in time.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov, ‘Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context’, CoRR
Original
2019
Closest in time.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’, in Proc. of NAACL-HLT 2019
2019
Closest in time.
Kalpit Dixit and Yaser Al-Onaizan, ‘Span-level model for relation extraction’, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
2019
Closest in time.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy, ‘SpanBERT: Improving Pre-training by Representing and Predicting Spans’, CoRR
Original
2019
Closest in time.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li, ‘Entity-Relation Extraction as Multi-Turn Question Answering’, in Proc. of ACL 2019
2019
Closest in time.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, ‘RoBERTa: A Robustly Optimized BERT Pretraining Approach’, CoRR
Original
2019
Closest in time.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi, ‘A General Framework for Information Extraction using Dynamic Span Graphs’, in Proc. of NAACL-HLT 2019
2019
Closest in time.
Dat Quoc Nguyen and Karin Verspoor, ‘End-to-end neural relation extraction using deep biaffine attention’, in Proc. of ECIR 2019
2019
Closest in time.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu, ‘MASS: Masked Sequence to Sequence Pre-training for Language Generation’, CoRR
Original
2019
Closest in time.
Tung Tran and Ramakanth Kavuluru, ‘Neural Metric Learning for Fast End-to-End Relation Extraction’, CoRR
Original
2019
Closest in time.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi, ‘Entity, Relation, and Event Extraction with Contextualized Span Representations’, ArXiv
Original
2019
Closest in time.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar, ‘Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers’, in Proc. of ACL 2019
2019
Closest in time.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin, ‘End-to-End Open-Domain Question Answering with BERTserini’, in Proc. of NAACL 2019 (Demonstrations)
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…