2020

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Li, Chunyuan, Gao, Xiang, Li, Yuan et al.

Understand

When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language.

In this paper, we propose the first large-scale language VAE model, Optimus.
A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks.
Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors.

Built on

Variational pretraining for semi-supervised text classification
Original
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A Smith. 2019 · 1906
Earlier work this paper cites.
Roberta: A robustly optimized BERT pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Ctrl: A conditional transformer language model for controllable generation
Original
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019 · 1909
Earlier work this paper cites.
Megatron-LM: Training multi-billion parameter language models using gpu model parallelism
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 1909
Earlier work this paper cites.
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Original
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910
Earlier work this paper cites.
Pre-train and plug-in: Flexible conditional text generation with variational auto-encoders
Original
Yu Duan, Jiaxin Pei, Canwen Xu, and Chenliang Li. 2019 · 1911
Earlier work this paper cites.
Building a large annotated corpus of english: The penn treebank
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993 · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jurgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
The information bottleneck method
Naftali Tishby, Fernando C Pereira, and William Bialek. 2000 · 2000
Earlier work this paper cites.
Texygen: A benchmarking platform for text generation models
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018 · 2000
Earlier work this paper cites.
Few-shot natural language generation for task-oriented dialog
Original
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020b · 2002
Earlier work this paper cites.
A batch normalized inference network keeps the kl vanishing away
Original
Qile Zhu, Wei Bi, Xiaojiang Liu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. 2020 · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 2005
Earlier work this paper cites.
Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model
Original
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020a · 2005
Earlier work this paper cites.
Visualizing data using t-SNE
Laurens van der Maaten and Geoffrey Hinton. 2008 · 2008
Earlier work this paper cites.
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010 · 2010
Earlier work this paper cites.
Auto-encoding variational bayes
Diederik P Kingma and Max Welling. 2013 · 2013
Earlier work this paper cites.
A systematic comparison of smoothing techniques for sentence-level Bleu
Boxing Chen and Colin Cherry. 2014 · 2014
Earlier work this paper cites.
Distributed representations of sentences and documents
Quoc Le and Tomas Mikolov. 2014 · 2014
Earlier work this paper cites.

Similar

Conditional generative adversarial nets
Original
Mehdi Mirza and Simon Osindero. 2014 · 2014
Cited alongside, same era.
Stochastic backpropagation and approximate inference in deep generative models
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014 · 2014
Cited alongside, same era.
A large annotated corpus for learning natural language inference
Original
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015 · 2015
Cited alongside, same era.
Importance weighted autoencoders
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015 · 2015
Cited alongside, same era.
Semi-supervised sequence learning
Andrew M Dai and Quoc V Le. 2015 · 2015
Cited alongside, same era.
Skip-thought vectors
Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
A hierarchical neural autoencoder for paragraphs and documents
Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015 · 2015
Cited alongside, same era.
Generating sentences from a continuous space
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016 · 2016
Cited alongside, same era.
Deep learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016 · 2016
Cited alongside, same era.
Elbo surgery: yet another way to carve up the variational evidence lower bound
Matthew D Hoffman and Matthew J Johnson. 2016 · 2016
Cited alongside, same era.
Improved variational inference with inverse autoregressive flow
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016 · 2016
Cited alongside, same era.
Adversarial autoencoders
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2016 · 2016
Cited alongside, same era.

Then

Isolating sources of disentanglement in VAEs
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. 2018 · 2018
Later among the works it cites.
Avoiding latent variable collapse with generative skip models
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. 2018 · 2018
Later among the works it cites.
A hierarchical latent structure for variational conversation modeling
Original
Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018 · 2018
Later among the works it cites.
Deep contextualized word representations
Original
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Towards text generation with adversarially learned neural outlines
Sandeep Subramanian, Sai Rajeswar Mudumba, Alessandro Sordoni, Adam Trischler, Aaron C Courville, and Chris Pal. 2018 · 2018
Later among the works it cites.
Adversarially regularized autoencoders
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2018 · 2018
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Unified language model pre-training for natural language understanding and generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019 · 2019
Later among the works it cites.
Implicit deep latent variable models for text generation
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019 · 2019
Later among the works it cites.
Cyclical annealing schedule: A simple approach to mitigating KL vanishing
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, Lawrence Carin, et al. 2019 · 2019
Later among the works it cites.
DialogWAE: Multimodal response generation with conditional wasserstein auto-encoder
Xiaodong Gu, Kyunghyun Cho, Jungwoo Ha, and Sunghun Kim. 2019 · 2019
Later among the works it cites.
Lagging inference networks and posterior collapse in variational autoencoders
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019 · 2019
Later among the works it cites.
A surprisingly effective fix for deep latent variable modeling of text
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019 · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Later among the works it cites.
Towards generating long and coherent text with multi-level latent variable models
Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. 2019 · 2019
Later among the works it cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019 · 2019
Later among the works it cites.
T-CVAE: Transformer-based conditioned variational autoencoder for story completion
Tianming Wang and Xiaojun Wan. 2019 · 2019
Later among the works it cites.
XLNet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019 · 2019
Later among the works it cites.
Defending against neural fake news
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…