2020

Residual Energy-Based Models for Text Generation

Deng, Yuntian, Bakhtin, Anton, Ott, Myle et al.

Understand

Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation.

The dominant parametric approach is based on locally normalized models which predict one word at a time.
While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process.
In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level.

Built on

Implicit generation and generalization in energy-based models
Original
Yilun Du and Igor Mordatch · 1903
Earlier work this paper cites.
A generalization of sampling without replacement from a finite universe
Daniel G. Horvitz and Donovan J. Thompson · 1952
Earlier work this paper cites.
Neural networks and physical systems with emergent collective computational abilities
John Hopfield · 1982
Earlier work this paper cites.
Une approche théorique de l’apprentissage connexionniste: Applications à la reconnaissance de la parole, 1991
Léon Bottou · 1991
Earlier work this paper cites.
Whole-sentence exponential language models: a vehicle for linguistic-statistical integration
Ronald Rosenfeld, Stanley F Chen, and Xiaojin Zhu · 2001
Earlier work this paper cites.
Training products of experts by minimizing contrastive divergence
Geoffrey E. Hinton · 2002
Earlier work this paper cites.
Energy-based models for sparse overcomplete representations
Y. W. Teh, M. Welling, S. Osindero, and Hinton G. E · 2003
Earlier work this paper cites.
Discriminative reranking for machine translation
Libin Shen, Anoop Sarkar, and Franz Josef Och · 2004
Earlier work this paper cites.
Exponential family harmoniums with an application to information retrieval
Max Welling, Michal Rosen-Zvi, and Geoffrey E. Hinton · 2005
Earlier work this paper cites.
A tutorial on energy-based learning
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu-Jie Huang · 2006
Earlier work this paper cites.
A unified energy-based framework for unsupervised learning
Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun · 2007
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Michael Gutmann and Aapo Hyvärinen · 2010
Earlier work this paper cites.
A practical guide to training restricted boltzmann machines
Geoffrey E Hinton · 2012
Earlier work this paper cites.
Monte Carlo theory, methods and examples
Art B. Owen · 2013
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Earlier work this paper cites.

Similar

Generative adversarial nets
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Cited alongside, same era.
Skip-thought vectors
Original
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, and Sanja Urtasun, Raquel an d Fidler · 2015
Cited alongside, same era.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015
Cited alongside, same era.
Trans-dimensional random fields for language modeling
Bin Wang, Zhijian Ou, and Zhiqiang Tan · 2015
Cited alongside, same era.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler · 2015
Cited alongside, same era.
Generating sentences from a continuous space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio · 2016
Cited alongside, same era.
Sequence level training with recurrent neural networks
Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba · 2016
Cited alongside, same era.
A theory of generative convnet
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu · 2016
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Language modeling with neural trans-dimensional random fields
Bin Wang and Zhijian Ou · 2017
Cited alongside, same era.
Learning trans-dimensional random fields with applications to language modeling
Bin Wang, Zhijian Ou, and Zhiqiang Tan · 2017
Cited alongside, same era.
Synthesizing dynamic patterns by spatial-temporal generative convnet
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu · 2017
Cited alongside, same era.

Then

Learning generative convnets via multi-grid modeling and sampling
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu · 2018
Later among the works it cites.
Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency
Zhuang Ma and Michael Collins · 2018
Later among the works it cites.
Debiasing evidence approximations: On importance-weighted autoencoders and jackknife variational inference
Sebastian Nowozin · 2018
Later among the works it cites.
Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimation
Bin Wang and Zhijian Ou · 2018
Later among the works it cites.
Learning neural trans-dimensional random field language models with noise-contrastive estimation
Bin Wang and Zhijian Ou · 2018
Later among the works it cites.
Learning descriptor networks for 3d shape synthesis and analysis
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu · 2018
Later among the works it cites.
Adversarially regularized autoencoders
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun · 2018
Later among the works it cites.
Real or fake? learning to discriminate machine from human generated text
Original
Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam · 2019
Later among the works it cites.
Bias correction of learned generative models using likelihood-free importance weighting
Original
Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon · 2019
Later among the works it cites.
The curious case of neural text degeneration
Original
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi · 2019
Later among the works it cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Later among the works it cites.
Learning non-convergent non-persistent short-run mcmc toward energy-based model
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu · 2019
Later among the works it cites.
Global autoregressive models for data-efficient sequence learning
Tetiana Parshakova, Jean-Marc Andreoli, and Marc Dymetman · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Learning energy-based spatial-temporal generative convnets for dynamic patterns
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…