2022

Knowledge Graph Generation From Text

Melnyk, Igor, Dognin, Pierre, Das, Payel

Understand

In this work we propose a novel end-to-end multi-stage Knowledge Graph (KG) generation system from textual inputs, separating the overall process into two stages.

The graph nodes are generated first using pretrained language model, followed by a simple edge construction head, enabling efficient KG extraction from the text.
For each stage we consider several architectural choices that can be used depending on the available training resources.
We evaluated the model on a recent WebNLG 2020 Challenge dataset, matching the state-of-the-art performance on text-to-RDF generation task, as well as on New York Times (NYT) and a large-scale TekGen datasets, showing strong overall performance, outperforming the existing baselines.

Built on

KG-BERT: BERT for Knowledge Graph Completion
Original
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019 · 1909
Earlier work this paper cites.
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 1910
Earlier work this paper cites.
End-to-end object detection with transformers
Original
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020 · 2005
Earlier work this paper cites.
CycleGT: Unsupervised graph-to-text and text-to-graph generation via cycle training
Original
Qipeng Guo, Zhijing Jin, Xipeng Qiu, W. Zhang, D. Wipf, and Zheng Zhang. 2020b · 2006
Earlier work this paper cites.
Investigating pretrained language models for graph-to-text generation
Original
Leonardo F. R. Ribeiro, Martin Schmitt, H. Schutze, and Iryna Gurevych. 2020 · 2007
Earlier work this paper cites.
Modeling relations and their mentions without labeled text
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010 · 2010
Earlier work this paper cites.
Autoprompt: Eliciting knowledge from language models with automatically generated prompts
Original
Taylor Shin, Yasaman Razeghi, IV RobertLLogan, Eric Wallace, and Sameer Singh. 2020 · 2010
Earlier work this paper cites.
Language models are open knowledge graphs
Original
C. Wang, Xiao Liu, and D. Song. 2020 · 2010
Earlier work this paper cites.
SemEval-2013 Task 9 : Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013)
Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo. 2013 · 2013
Earlier work this paper cites.
The Stanford CoreNLP natural language processing toolkit
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014 · 2014
Earlier work this paper cites.
Commonsense Knowledge Base Completion
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016 · 2016
Earlier work this paper cites.

Similar

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
GraphRNN: Generating realistic graphs with deep auto-regressive models
Jiaxuan You, Rex Ying, Xiang Ren, William L. Hamilton, and J. Leskovec. 2018 · 2018
Cited alongside, same era.
Extracting relational facts by an end-to-end neural model with copy mechanism
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018 · 2018
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
How can we know what language models know?
Zhengbao Jiang, F. F. Xu, J. Araki, and Graham Neubig. 2019 · 2019
Cited alongside, same era.
Language models as knowledge bases?
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, A. Bakhtin, Yuxiang Wu, Alexander H. Miller, and S. Riedel. 2019 · 2019
Cited alongside, same era.
Machine translation aided bilingual data-to-text generation and semantic parsing
Oshin Agarwal, Mihir Kale, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2020 · 2020
Cited alongside, same era.
DualTKB: A Dual Learning Bridge between Text and Knowledge Base
Pierre Dognin, Igor Melnyk, Inkit Padhi, Cicero Nogueira dos Santos, and Payel Das. 2020 · 2020
Cited alongside, same era.
The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)
Thiago Castro Ferreira, Claire Gardent, N. Ilinykh, C. Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020 · 2020
Cited alongside, same era.
Leveraging large pretrained models for WebNLG 2020
Xintong Li, Aleksandre Maskharashvili, S. Stevens-Guille, and Michael White. 2020 · 2020
Cited alongside, same era.
P2: A plan-and-pretrain approach for knowledge graph-to-text generation
Qipeng Guo, Zhijing Jin, Ning Dai, Xipeng Qiu, Xiangyang Xue, David Wipf, and Zheng Zhang. 2020a
Cited in the paper.

Then

Focal loss for dense object detection
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2020 · 2020
Later among the works it cites.
Commonsense Knowledge Base Completion with Structural and Semantic Context
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020 · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Later among the works it cites.
How much knowledge can you pack into the parameters of a language model?
Adam Roberts, Colin Raffel, and Noam M. Shazeer. 2020 · 2020
Later among the works it cites.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020 · 2020
Later among the works it cites.
Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021 · 2021
Later among the works it cites.
Regen: Reinforcement learning for text and knowledge base generation using pretrained language models
Original
Pierre L. Dognin, Inkit Padhi, Igor Melnyk, and Payel Das. 2021 · 2021
Later among the works it cites.
Prefix-tuning: Optimizing continuous prompts for generation
Original
Xiang Lisa Li and Percy Liang. 2021 · 2021
Later among the works it cites.
Unified structure generation for universal information extraction
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022 · 2022
Closest in time.
Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…