Fetching the paper…

Reading the bibliography…

2019

Leveraging Lead Bias for Zero-shot Abstractive News Summarization

Zhu, Chenguang, Yang, Ziyi, Gmyr, Robert et al.

Original
PDF
Post on X

Understand

A typical journalistic convention in news articles is to deliver the most salient information in the beginning, also known as the lead bias.

While this phenomenon can be exploited in generating a summary, it has a detrimental effect on teaching a model to discriminate and extract important information in general.
We propose that this lead bias can be leveraged in our favor in a simple and effective way to pre-train abstractive news summarization models on large-scale unlabeled news corpora: predicting the leading sentences using the rest of an article.
We collect a massive news corpus and conduct data cleaning and filtering via statistical analysis.

Leveraging Lead Bias for Zero-shot Abstractive News Summarization · Around

Built on

TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising
Original
Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. 2020 · 2001
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
Pre-training for Abstractive Document Summarization by Reinstating Source Text
Original
Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, and Ming Zhou. 2020 · 2004
Earlier work this paper cites.
DUC in context
Paul Over, Hoa Dang, and Donna Harman. 2007 · 2007
Earlier work this paper cites.
The new york times annotated corpus
Evan Sandhaus. 2008 · 2008
Earlier work this paper cites.
Teaching machines to read and comprehend
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015
Earlier work this paper cites.
A neural attention model for abstractive sentence summarization
Original
Alexander M Rush, Sumit Chopra, and Jason Weston. 2015 · 2015
Earlier work this paper cites.
A deep reinforced model for abstractive summarization
Original
Romain Paulus, Caiming Xiong, and Richard Socher. 2017 · 2017
Earlier work this paper cites.
Get to the point: Summarization with pointer-generator networks
Original
Abigail See, Peter J Liu, and Christopher D Manning. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
Original
Eric Chu and Peter J Liu. 2018 · 2018
Earlier work this paper cites.

Similar

Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018
Cited alongside, same era.
Unsupervised Sentence Compression using Denoising Auto-Encoders
Original
Thibault Févry and Jason Phang. 2018 · 2018
Cited alongside, same era.
Content selection in deep learning models of summarization
Original
Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018 · 2018
Cited alongside, same era.
Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Original
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018 · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training

Then

Importance of Copying Mechanism for News Headline Generation
Original
Ilya Gusev. 2019 · 2019
Closest in time.
Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019 · 2019
Closest in time.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 2019
Closest in time.
On the Variance of the Adaptive Learning Rate and Beyond
Original
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019b · 2019
Closest in time.
SummAE: Zero-shot abstractive text summarization using length-agnostic auto-encoders

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018

Cited alongside, same era.

Learning to encode text as human-readable summaries using generative adversarial networks

Yau-Shian Wang and Hung-Yi Lee. 2018 · 2018

Cited alongside, same era.

SEQˆ 3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019a · 2019

Cited alongside, same era.

SEQˆ 3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019b · 2019

Cited alongside, same era.

Unified Language Model Pre-training for Natural Language Understanding and Generation

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019 · 2019

Cited alongside, same era.

Countering the Effects of Lead Bias in News Summarization via Multi-stage Training and Auxiliary Losses

Matt Grenander, Yue Dong, Jackie C.K. Cheung, and Annie Louis. 2019 · 2019

Cited alongside, same era.

Peter J Liu, Yu-An Chung, and Jie Ren. 2019a · 2019

Closest in time.

Text Summarization with Pretrained Encoders

Yang Liu and Mirella Lapata. 2019 · 2019

Closest in time.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 2019

Closest in time.

Pegasus: Pre-training with extracted gap-sentences for abstractive summarization

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2019 · 2019

Closest in time.

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020 · 2020

Closest in time.

Boosting factual correctness of abstractive summarization with knowledge graph

Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020 · 2020

Closest in time.