2017

Efficient Vector Representation for Documents through Corruption

Chen, Minmin

Understand

We present an efficient document representation learning framework, Document Vector through Corruption (Doc2VecC).

Doc2VecC represents each document as a simple average of word embeddings.
It ensures a representation generated as such captures the semantic meanings of the document during learning.
A corruption model is included, which introduces a data-dependent regularization that favors informative or rare words while forcing the embeddings of common and non-discriminative ones to be close to zero.

Built on

Term-weighting approaches in automatic text retrieval
Salton, G. and Buckley, C. (1988) · 1988
Earlier work this paper cites.
Indexing by latent semantic analysis
Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. (1990) · 1990
Earlier work this paper cites.
Latent dirichlet allocation
Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003) · 2003
Earlier work this paper cites.
Liblinear: A library for large linear classification
Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., and Lin, C.-J. (2008) · 2008
Earlier work this paper cites.
Visualizing data using t-sne
Maaten, L. v. d. and Hinton, G. (2008) · 2008
Earlier work this paper cites.
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008) · 2008
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., and Khudanpur, S. (2010) · 2010
Earlier work this paper cites.
Composition in distributional models of semantics
Mitchell, J. and Lapata, M. (2010) · 2010
Earlier work this paper cites.
Estimating linear models for compositional distributional semantics
Zanzotto, F. M., Korkontzelos, I., Fallucchi, F., and Manandhar, S. (2010) · 2010
Earlier work this paper cites.
Domain adaptation for large-scale sentiment classification: A deep learning approach
Glorot, X., Bordes, A., and Bengio, Y. (2011) · 2011
Earlier work this paper cites.
Learning word vectors for sentiment analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. (2011) · 2011
Earlier work this paper cites.
Compositional matrix-space models for sentiment analysis
Yessenalina, A. and Cardie, C. (2011) · 2011
Earlier work this paper cites.

Similar

Marginalized denoising autoencoders for domain adaptation
Original
Chen, M., Xu, Z., Weinberger, K., and Sha, F. (2012) · 2012
Cited alongside, same era.
Improving word representations via global context and multiple word prototypes
Huang, E. H., Socher, R., Manning, C. D., and Ng, A. Y. (2012) · 2012
Cited alongside, same era.
Baselines and bigrams: Simple, good sentiment and topic classification
Wang, S. and Manning, C. D. (2012) · 2012
Cited alongside, same era.
Language modeling for information retrieval
Croft, B. and Lafferty, J. (2013) · 2013
Cited alongside, same era.
Multi-step regression learning for compositional distributional semantics
Original
Grefenstette, E., Dinu, G., Zhang, Y.-Z., Sadrzadeh, M., and Baroni, M. (2013) · 2013
Cited alongside, same era.
Distributed representations of words and phrases and their compositionality
Mikolov, T. and Dean, J. (2013) · 2013
Cited alongside, same era.
Recursive deep models for semantic compositionality over a sentiment treebank
Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. (2013) · 2013
Cited alongside, same era.
Learning with marginalized corrupted features
Van Der Maaten, L., Chen, M., Tyree, S., and Weinberger, K. Q. (2013) · 2013
Cited alongside, same era.
Dropout training as adaptive regularization
Wager, S., Wang, S., and Liang, P. S. (2013) · 2013
Cited alongside, same era.
Marginalized denoising auto-encoders for nonlinear representations
Chen, M., Weinberger, K. Q., Sha, F., and Bengio, Y. (2014) · 2014
Cited alongside, same era.
Distributed representations of sentences and documents
Le, Q. V. and Mikolov, T. (2014) · 2014
Cited alongside, same era.
Efficient estimation of word representations in vector space
Original
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a)
Cited in the paper.

Then

Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment
Marelli, M., Bentivogli, L., Baroni, M., Bernardi, R., Menini, S., and Zamparelli, R. (2014) · 2014
Later among the works it cites.
Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews
Original
Mesnil, G., Mikolov, T., Ranzato, M., and Bengio, Y. (2014) · 2014
Later among the works it cites.
Grounded compositional semantics for finding and describing images with sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., and Ng, A. Y. (2014) · 2014
Later among the works it cites.
Semi-supervised sequence learning
Dai, A. M. and Le, Q. V. (2015) · 2015
Later among the works it cites.
Document embedding with paragraph vectors
Original
Dai, A. M., Olah, C., and Le, Q. V. (2015) · 2015
Later among the works it cites.
Character-aware neural language models
Original
Kim, Y., Jernite, Y., Sontag, D., and Rush, A. M. (2015) · 2015
Later among the works it cites.
Skip-thought vectors
Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015) · 2015
Later among the works it cites.
From word embeddings to document distances
Kusner, M. J., Sun, Y., Kolkin, N. I., and Weinberger, K. Q. (2015) · 2015
Later among the works it cites.
Improved semantic representations from tree-structured long short-term memory networks
Original
Tai, K. S., Socher, R., and Manning, C. D. (2015) · 2015
Later among the works it cites.
Text understanding from scratch
Original
Zhang, X. and LeCun, Y. (2015) · 2015
Later among the works it cites.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Original
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015) · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…