2020

SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization

Zhao, Yao, Saleh, Mohammad, Liu, Peter J.

Understand

Most prior work in the sequence-to-sequence paradigm focused on datasets with input sequence lengths in the hundreds of tokens due to the computational constraints of common RNN and Transformer architectures.

In this paper, we study long-form abstractive text summarization, a sequence-to-sequence setting with input sequence lengths up to 100,000 tokens and output sequence lengths up to 768 tokens.
We propose SEAL, a Transformer-based model, featuring a new encoder-decoder attention that dynamically extracts/selects input snippets to sparsely attend to for each output segment.
Using only the original documents and summaries, we derive proxy labels that provide weak supervision for extractive layers simultaneously with regular supervision from abstractive summaries.

Built on

Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J. Williams · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
ROUGE: A package for automatic evaluation of summaries
Chin-Yew Lin · 2004
Earlier work this paper cites.
Guiding generation for abstractive text summarization based on key information guide network
Chenliang Li, Weiran Xu, Si Li, and Sheng Gao · 2009
Earlier work this paper cites.
Automatically generating Wikipedia articles: A structure-aware approach
Christina Sauper and Regina Barzilay · 2009
Earlier work this paper cites.
Sequence to sequence learning with neural networks, 2014
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le · 2014
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling, 2014
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio · 2014
Earlier work this paper cites.
A neural attention model for abstractive sentence summarization
Alexander M. Rush, Sumit Chopra, and Jason Weston · 2015
Earlier work this paper cites.
Pointer networks
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly · 2015
Earlier work this paper cites.
Teaching machines to read and comprehend, 2015
Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom · 2015
Earlier work this paper cites.
Improving neural language models with a continuous cache, 2016
Edouard Grave, Armand Joulin, and Nicolas Usunier · 2016
Earlier work this paper cites.
Rationalizing neural predictions
Tao Lei, Regina Barzilay, and Tommi Jaakkola · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean · 2016
Earlier work this paper cites.

Similar

Get to the point: Summarization with pointer-generator networks
Abigail See, Peter J. Liu, and Christopher D. Manning · 2017
Cited alongside, same era.
Attention is all you need, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Summarunner: A recurrent neural network based sequence model for extractive summarization of documents
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou · 2017
Cited alongside, same era.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean · 2017
Cited alongside, same era.
Hard mixtures of experts for large scale weakly supervised vision
Sam Gross, Marc’Aurelio Ranzato, and Arthur Szlam · 2017
Cited alongside, same era.
Gated self-matching networks for reading comprehension and question answering
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou · 2017
Cited alongside, same era.
A discourse-aware attention model for abstractive summarization of long documents
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian · 2018
Cited alongside, same era.
Generating wikipedia by summarizing long sequences, 2018
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer · 2018
Cited alongside, same era.
Bottom-up abstractive summarization
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush · 2018
Cited alongside, same era.
Fast abstractive summarization with reinforce-selected sentence rewriting
Yen-Chun Chen and Mohit Bansal · 2018
Cited alongside, same era.
Adafactor: Adaptive learning rates with sublinear memory cost, 2018
Noam Shazeer and Mitchell Stern · 2018
Cited alongside, same era.
Text summarization with pretrained encoders
Yang Liu and Mirella Lapata · 2019
Cited alongside, same era.

Then

Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
On extractive and abstractive neural document summarization with transformer language models, 2019
Sandeep Subramanian, Raymond Li, Jonathan Pilault, and Christopher Pal · 2019
Later among the works it cites.
Informative and controllable opinion summarization, 2019
Reinald Kim Amplayo and Mirella Lapata · 2019
Later among the works it cites.
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 2019
Later among the works it cites.
Transformer-xl: Attentive language models beyond a fixed-length context
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov · 2019
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2019
Later among the works it cites.
Mass: Masked sequence to sequence pre-training for language generation
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu · 2019
Later among the works it cites.
Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu · 2019
Later among the works it cites.
Unsupervised paraphrasing without translation
Aurko Roy and David Grangier · 2019
Later among the works it cites.
Extractive summarization of long documents by combining global and local context, 2019
Wen Xiao and Giuseppe Carenini · 2019
Later among the works it cites.
Reformer: The efficient transformer, 2020
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya · 2020
Closest in time.
Efficient content-based sparse attention with routing transformers, 2020
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…