2020

Exploring Controllable Text Generation Techniques

Prabhumoye, Shrimai, Black, Alan W, Salakhutdinov, Ruslan

Understand

Neural controllable text generation is an important area gaining attention due to its plethora of applications.

Although there is a large body of prior work in controllable text generation, there is no unifying theme.
In this work, we provide a new schema of the pipeline of the generation process by classifying it into five modules.
The control of attributes in the generation process requires modification of these modules.

Built on

Unsupervised controllable text generation with global variation discovery and disentanglement
Original
Peng Xu, Yanshuai Cao, and Jackie Chi Kit Cheung. 2019 · 1905
Earlier work this paper cites.
What makes a good story? designing composite rewards for visual storytelling
Original
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2019a · 1909
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
Plug and play language models: a simple approach to controlled text generation
Original
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019 · 1912
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
Ronald J Williams and David Zipser. 1989 · 1989
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams. 1992 · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Building Natural Language Generation Systems
Ehud Reiter and Robert Dale. 2000 · 2000
Earlier work this paper cites.
I love your chain mail! making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
Original
Shrimai Prabhumoye, Margaret Li, Jack Urbanek, Emily Dinan, Douwe Kiela, Jason Weston, and Arthur Szlam. 2020 · 2002
Earlier work this paper cites.
Learning phrase representations using rnn encoder–decoder for statistical machine translation
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Auto-encoding variational bayes
Diederik P Kingma and Max Welling. 2014 · 2014
Earlier work this paper cites.
Neural machine translation by jointly learning to align and translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
Effective approaches to attention-based neural machine translation
Original
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015 · 2015
Earlier work this paper cites.
Semantically conditioned LSTM-based natural language generation for spoken dialogue systems
Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015 · 2015
Earlier work this paper cites.
Generating sentences from a continuous space
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016 · 2016
Earlier work this paper cites.
Incorporating side information into recurrent neural network language models
Cong Duy Vu Hoang, Trevor Cohn, and Gholamreza Haffari. 2016 · 2016
Earlier work this paper cites.
Globally coherent text generation with neural checklist models
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016 · 2016
Earlier work this paper cites.
Gans for sequences of discrete elements with the gumbel-softmax distribution
Original
Matt J Kusner and José Miguel Hernández-Lobato. 2016 · 2016
Earlier work this paper cites.
Professor forcing: A new algorithm for training recurrent networks
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016 · 2016
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a · 2016
Earlier work this paper cites.
Stylenet: Generating attractive visual captions with styles
Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017 · 2017
Earlier work this paper cites.
Toward controlled generation of text
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017 · 2017
Earlier work this paper cites.
Definition modeling: Learning to define word embeddings in natural language
Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017 · 2017
Earlier work this paper cites.

Similar

Get to the point: Summarization with pointer-generator networks
Abigail See, Peter J Liu, and Christopher D Manning. 2017 · 2017
Cited alongside, same era.
Style transfer from non-parallel text by cross-alignment
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Neural text generation: A practical guide
Original
Ziang Xie. 2017 · 2017
Cited alongside, same era.
Seqgan: Sequence generative adversarial nets with policy gradient
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017 · 2017
Cited alongside, same era.
Wizard of wikipedia: Knowledge-powered conversational agents
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018 · 2018
Cited alongside, same era.
Hierarchical neural story generation
Angela Fan, Mike Lewis, and Yann Dauphin. 2018 · 2018
Cited alongside, same era.
Style transfer in text: Exploration and evaluation
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018 · 2018
Cited alongside, same era.
A knowledge-grounded neural conversation model
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018 · 2018
Cited alongside, same era.
Learning to write with cooperative discriminators
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018 · 2018
Cited alongside, same era.
On unifying deep generative models
Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. 2018 · 2018
Cited alongside, same era.
Improving neural abstractive document summarization with structural regularization
Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018 · 2018
Cited alongside, same era.

Then

A dataset for document grounded conversations
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018 · 2018
Later among the works it cites.
“my way of telling a story”: Persona based grounded story generation
Khyathi Chandu, Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. 2019 · 2019
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Reinforcement learning based text style transfer without parallel training corpus
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019 · 2019
Later among the works it cites.
Non-autoregressive neural machine translation with enhanced decoder input
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019 · 2019
Later among the works it cites.
Maximizing stylistic control and semantic accuracy in nlg: Personality variation and discourse contrast
Vrindavan Harrison, Lena Reed, Shereen Oraby, and Marilyn Walker. 2019 · 2019
Later among the works it cites.
Hierarchically structured reinforcement learning for topically coherent visual story generation
Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang, and Xiaodong He. 2019 · 2019
Later among the works it cites.
Disentangled representation learning for non-parallel text style transfer
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019 · 2019
Later among the works it cites.
Importance of search and evaluation strategies in neural dialogue modeling
Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019 · 2019
Later among the works it cites.
Towards content transfer through grounded text generation
Shrimai Prabhumoye, Chris Quirk, and Michel Galley. 2019 · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Later among the works it cites.
Adversarial decomposition of text representation
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019 · 2019
Later among the works it cites.
“transforming” delete, retrieve, generate approach for controlled text style transfer
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019 · 2019
Later among the works it cites.
Target-guided open-domain conversation
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019 · 2019
Later among the works it cites.
Learning to speak and act in a fantasy text adventure game
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019 · 2019
Later among the works it cites.
Topic-guided variational auto-encoder for text generation
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019b · 2019
Later among the works it cites.
Non-monotonic sequential text generation
Sean Welleck, Kianté Brantley, Hal Daumé Iii, and Kyunghyun Cho. 2019 · 2019
Later among the works it cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 · 2019
Later among the works it cites.
Plan-and-write: Towards better automatic storytelling
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019 · 2019
Later among the works it cites.
Structsum: Incorporating latent and explicit sentence dependencies for single document summarization
Original
Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. 2020 · 2020
Closest in time.
The curious case of neural text degeneration
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020 · 2020
Closest in time.
Politeness transfer: A tag and generate approach
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020 · 2020
Closest in time.
Neural text generation with unlikelihood training
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020 · 2020
Closest in time.
Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history
Yiheng Zhou, Yulia Tsvetkov, Alan W Black, and Zhou Yu. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…