2017

Learning to Generate Reviews and Discovering Sentiment

Radford, Alec, Jozefowicz, Rafal, Sutskever, Ilya

Understand

We explore the properties of byte-level recurrent language models.

When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts.
Specifically, we find a single unit which performs sentiment analysis.
These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank.

Built on

Building a large annotated corpus of english: The penn treebank
Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice · 1993
Earlier work this paper cites.
Sparse coding with an overcomplete basis set: A strategy employed by v1?
Olshausen, Bruno A and Field, David J · 1997
Earlier work this paper cites.
Latent dirichlet allocation
Blei, David M, Ng, Andrew Y, and Jordan, Michael I · 2003
Earlier work this paper cites.
Utf-8, a transformation format of iso 10646
Yergeau, Francois · 2003
Earlier work this paper cites.
Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources
Dolan, Bill, Quirk, Chris, and Brockett, Chris · 2004
Earlier work this paper cites.
Mining and summarizing customer reviews
Hu, Minqing and Liu, Bing · 2004
Earlier work this paper cites.
Feature selection, l 1 vs. l 2 regularization, and rotational invariance
Ng, Andrew Y · 2004
Earlier work this paper cites.
A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts
Pang, Bo and Lee, Lillian · 2004
Earlier work this paper cites.
Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
Pang, Bo and Lee, Lillian · 2005
Earlier work this paper cites.
Annotating expressions of opinions and emotions in language
Wiebe, Janyce, Wilson, Theresa, and Cardie, Claire · 2005
Earlier work this paper cites.
Reducing the dimensionality of data with neural networks
Hinton, Geoffrey E and Salakhutdinov, Ruslan R · 2006
Earlier work this paper cites.
The human knowledge compression contest. 2006
Hutter, Marcus · 2006
Earlier work this paper cites.
Unsupervised learning of invariant feature hierarchies with applications to object recognition
Huang, Fu Jie, Boureau, Y-Lan, LeCun, Yann, et al · 2007
Earlier work this paper cites.
Extracting and composing robust features with denoising autoencoders
Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, Pierre-Antoine · 2008
Earlier work this paper cites.
An analysis of single-layer networks in unsupervised feature learning
Coates, Adam, Lee, Honglak, and Ng, Andrew Y · 2010
Earlier work this paper cites.
Natural language processing (almost) from scratch
Collobert, Ronan, Weston, Jason, Bottou, Léon, Karlen, Michael, Kavukcuoglu, Koray, and Kuksa, Pavel · 2011
Earlier work this paper cites.
Learning word vectors for sentiment analysis
Maas, Andrew L, Daly, Raymond E, Pham, Peter T, Huang, Dan, Ng, Andrew Y, and Potts, Christopher · 2011
Earlier work this paper cites.
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups
Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E · 2012
Earlier work this paper cites.

Similar

Re-examining machine translation metrics for paraphrase identification
Madnani, Nitin, Tetreault, Joel, and Chodorow, Martin · 2012
Cited alongside, same era.
Baselines and bigrams: Simple, good sentiment and topic classification
Wang, Sida and Manning, Christopher D · 2012
Cited alongside, same era.
Representation learning: A review and new perspectives
Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal · 2013
Cited alongside, same era.
Building high-level features using large scale unsupervised learning
Le, Quoc V · 2013
Cited alongside, same era.
Linguistic regularities in continuous space word representations
Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey · 2013
Cited alongside, same era.
Recursive deep models for semantic compositionality over a sentiment treebank
Socher, Richard, Perelygin, Alex, Wu, Jean Y, Chuang, Jason, Manning, Christopher D, Ng, Andrew Y, Potts, Christopher, et al · 2013
Cited alongside, same era.
A fast and accurate dependency parser using neural networks
Chen, Danqi and Manning, Christopher D · 2014
Cited alongside, same era.
Convolutional neural networks for sentence classification
Original
Kim, Yoon · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, Diederik and Ba, Jimmy · 2014
Cited alongside, same era.
Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment
Marelli, Marco, Bentivogli, Luisa, Baroni, Marco, Bernardi, Raffaella, Menini, Stefano, and Zamparelli, Roberto · 2014
Cited alongside, same era.
Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews
Original
Mesnil, Grégoire, Mikolov, Tomas, Ranzato, Marc’Aurelio, and Bengio, Yoshua · 2014
Cited alongside, same era.
Learning and transferring mid-level image representations using convolutional neural networks
Oquab, Maxime, Bottou, Leon, Laptev, Ivan, and Sivic, Josef · 2014
Cited alongside, same era.

Then

Inferring networks of substitutable and complementary products
McAuley, Julian, Pandey, Rahul, and Leskovec, Jure · 2015
Later among the works it cites.
Improved semantic representations from tree-structured long short-term memory networks
Original
Tai, Kai Sheng, Socher, Richard, and Manning, Christopher D · 2015
Later among the works it cites.
Towards universal paraphrastic sentence embeddings
Original
Wieting, John, Bansal, Mohit, Gimpel, Kevin, and Livescu, Karen · 2015
Later among the works it cites.
Character-level convolutional networks for text classification
Zhang, Xiang, Zhao, Junbo, and LeCun, Yann · 2015
Later among the works it cites.
Self-adaptive hierarchical sentence model
Original
Zhao, Han, Lu, Zhengdong, and Poupart, Pascal · 2015
Later among the works it cites.
Layer normalization
Original
Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E · 2016
Later among the works it cites.
Topicrnn: A recurrent neural network with long-range semantic dependency
Original
Dieng, Adji B, Wang, Chong, Gao, Jianfeng, and Paisley, John · 2016
Later among the works it cites.
Deep learning
Goodfellow, Ian, Bengio, Yoshua, and Courville, Aaron · 2016
Later among the works it cites.
Learning distributed representations of sentences from unlabelled data
Original
Hill, Felix, Cho, Kyunghyun, and Korhonen, Anna · 2016
Later among the works it cites.
Exploring the limits of language modeling
Original
Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui · 2016
Later among the works it cites.
Multiplicative lstm for sequence modelling
Original
Krause, Ben, Lu, Liang, Murray, Iain, and Renals, Steve · 2016
Later among the works it cites.
Understanding neural networks through representation erasure
Original
Li, Jiwei, Monroe, Will, and Jurafsky, Dan · 2016
Later among the works it cites.
Adversarial training methods for semi-supervised text classification
Original
Miyato, Takeru, Dai, Andrew M, and Goodfellow, Ian · 2016
Later among the works it cites.
Neural semantic encoders
Original
Munkhdalai, Tsendsuren and Yu, Hong · 2016
Later among the works it cites.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Salimans, Tim and Kingma, Diederik P · 2016
Later among the works it cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V, Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, et al · 2016
Later among the works it cites.
Learning to generate product reviews from attributes
Dong, Li, Huang, Shaohan, Wei, Furu, Lapata, Mirella, Zhou, Ming, and Ke, Xu · 2017
Closest in time.
Controllable text generation
Original
Hu, Zhiting, Yang, Zichao, Liang, Xiaodan, Salakhutdinov, Ruslan, and Xing, Eric P · 2017
Closest in time.
Deep learning with dynamic computation graphs
Original
Looks, Moshe, Herreshoff, Marcello, Hutchins, DeLesley, and Norvig, Peter · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…