2020

A Theoretical Analysis of the Repetition Problem in Text Generation

Fu, Zihao, Lam, Wai, So, Anthony Man-Cho et al.

Understand

Text generation tasks, including translation, summarization, language models, and etc.

see rapid growth during recent years.
Despite the remarkable achievements, the repetition problem has been observed in nearly all text generation models undermining the generation performance extensively.
To solve the repetition problem, many methods have been proposed, but there is no existing theoretical analysis to show why this problem happens and how it is resolved.

Built on

A learning algorithm for Boltzmann machines
Ackley, D. H.; Hinton, G. E.; and Sejnowski, T. J. 1985 · 1985
Earlier work this paper cites.
A Gersgorin-type lower bound for the smallest singular value
Johnson, C. R. 1989 · 1989
Earlier work this paper cites.
BLEU: a method for automatic evaluation of machine translation
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002 · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
Lin, C.-Y. 2004 · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020 · 2005
Earlier work this paper cites.
A Theoretical Analysis of the Repetition Problem in Text Generation
Original
Fu, Z.; Lam, W.; So, A. M.-C.; and Shi, B. 2020b · 2012
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Original
Bahdanau, D.; Cho, K.; and Bengio, Y. 2014 · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014 · 2014
Earlier work this paper cites.
Effective Approaches to Attention-based Neural Machine Translation
Luong, T.; Pham, H.; and Manning, C. D. 2015 · 2015
Earlier work this paper cites.
Neural Text Generation from Structured Data with Application to the Biography Domain
Lebret, R.; Grangier, D.; and Auli, M. 2016 · 2016
Earlier work this paper cites.
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R.; Zhou, B.; dos Santos, C.; Guulçehre, Ç.; and Xiang, B. 2016 · 2016
Earlier work this paper cites.

Similar

Neural Machine Translation of Rare Words with Subword Units
Sennrich, R.; Haddow, B.; and Birch, A. 2016 · 2016
Cited alongside, same era.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; et al. 2016 · 2016
Cited alongside, same era.
Controlling Linguistic Style Aspects in Neural Language Generation
Ficler, J.; and Goldberg, Y. 2017 · 2017
Cited alongside, same era.
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G.; Kim, Y.; Deng, Y.; Senellart, J.; and Rush, A. 2017 · 2017
Cited alongside, same era.
Pointer Sentinel Mixture Models
Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2017 · 2017
Cited alongside, same era.
The E2E Dataset: New Challenges For End-to-End Generation
Novikova, J.; Dušek, O.; and Rieser, V. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017 · 2017
Cited alongside, same era.
Challenges in Data-to-Document Generation
Wiseman, S.; Shieber, S.; and Rush, A. 2017 · 2017
Cited alongside, same era.
The Missing Representation in Neural Language Models
Choi, Y. 2018 · 2018
Cited alongside, same era.
Hierarchical Neural Story Generation
Fan, A.; Lewis, M.; and Dauphin, Y. 2018 · 2018
Cited alongside, same era.
Dynamic topic tracker for kb-to-text generation
Fu, Z.; Bing, L.; Lam, W.; and Jameel, S. 2020a
Cited in the paper.

Then

Improving Language Understanding by Generative Pre-Training
Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018 · 2018
Later among the works it cites.
Deconstructing bert: Distilling 6 patterns from 100 million parameters
Vig, J. 2018 · 2018
Later among the works it cites.
Learning Neural Templates for Text Generation
Wiseman, S.; Shieber, S.; and Rush, A. 2018 · 2018
Later among the works it cites.
Language GANs Falling Short
Caccia, M.; Caccia, L.; Fedus, W.; Larochelle, H.; Pineau, J.; and Charlin, L. 2019 · 2019
Later among the works it cites.
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; and Auli, M. 2019 · 2019
Later among the works it cites.
Language Models are Unsupervised Multitask Learners
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019 · 2019
Later among the works it cites.
Open Domain Event Text Generation
Fu, Z.; Bing, L.; and Lam, W. 2020 · 2020
Closest in time.
Partially-Aligned Data-to-Text Generation with Distant Supervision
Fu, Z.; Shi, B.; Lam, W.; Bing, L.; and Liu, Z. 2020c · 2020
Closest in time.
The Curious Case of Neural Text Degeneration
Holtzman, A.; Buys, J.; Du, L.; Forbes, M.; and Choi, Y. 2020 · 2020
Closest in time.
Neural Text Generation With Unlikelihood Training
Welleck, S.; Kulikov, I.; Roller, S.; Dinan, E.; Cho, K.; and Weston, J. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…