2016

Semantic Compositional Networks for Visual Captioning

Gan, Zhe, Gan, Chuang, He, Xiaodong et al.

Understand

A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network.

The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices.
The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag.
In addition to captioning images, we also extend the SCN to generate captions for video clips.

Built on

Finding structure in time
J. L. Elman · 1990
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
C.-Y. Lin · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
S. Banerjee and A. Lavie · 2005
Earlier work this paper cites.
Unsupervised learning of image transformations
R. Memisevic and G. Hinton · 2007
Earlier work this paper cites.
Factored conditional restricted boltzmann machines for modeling motion style
G. W. Taylor and G. E. Hinton · 2009
Earlier work this paper cites.
Collecting highly parallel data for paraphrase evaluation
D. L. Chen and W. B. Dolan · 2011
Earlier work this paper cites.
Generating text with recurrent neural networks
I. Sutskever, J. Martens, and G. E. Hinton · 2011
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean · 2013
Earlier work this paper cites.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Earlier work this paper cites.
Large-scale video classification with convolutional neural networks
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei · 2014
Earlier work this paper cites.
Multimodal neural language models
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2014
Earlier work this paper cites.
Unifying visual-semantic embeddings with multimodal neural language models
Original
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2014
Earlier work this paper cites.
A multiplicative model for learning distributed text-based attribute representations
R. Kiros, R. Zemel, and R. R. Salakhutdinov · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Grounded compositional semantics for finding and describing images with sentences
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng · 2014
Earlier work this paper cites.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. V. Le · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier · 2014
Earlier work this paper cites.

Similar

Recurrent neural network regularization
Original
W. Zaremba, I. Sutskever, and O. Vinyals · 2014
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
D. Bahdanau, K. Cho, and Y. Bengio · 2015
Cited alongside, same era.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick · 2015
Cited alongside, same era.
Mind’s eye: A recurrent visual representation for image caption generation
X. Chen and C. Lawrence Zitnick · 2015
Cited alongside, same era.
Language models for image captioning: The quirks and what works
J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell · 2015
Cited alongside, same era.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Cited alongside, same era.
From captions to visual concepts and back
H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, et al · 2015
Cited alongside, same era.
Guiding long-short term memory for image caption generation
X. Jia, E. Gavves, B. Fernando, and T. Tuytelaars · 2015
Cited alongside, same era.
Aligning where to see and what to tell: image caption with region-based attention and scene factorization
Original
J. Jin, K. Fu, R. Cui, F. Sha, and C. Zhang · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
D. Kingma and J. Ba · 2015
Cited alongside, same era.
Deep captioning with multimodal recurrent neural networks (m-rnn)
J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille · 2015
Cited alongside, same era.

Then

Deep compositional captioning: Describing novel object categories without paired training data
L. Anne Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell · 2016
Closest in time.
Delving deeper into convolutional networks for learning video representations
N. Ballas, L. Yao, C. Pal, and A. Courville · 2016
Closest in time.
Early embedding and late reranking for video captioning
J. Dong, X. Li, W. Lan, Y. Huo, and C. G. Snoek · 2016
Closest in time.
Learning attributes equals multi-source domain generalization
C. Gan, T. Yang, and B. Gong · 2016
Closest in time.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Closest in time.
Attention correctness in neural image captioning
Original
C. Liu, J. Mao, F. Sha, and A. Yuille · 2016
Closest in time.
Jointly modeling embedding and translation to bridge video and language
Y. Pan, T. Mei, T. Yao, H. Li, and Y. Rui · 2016
Closest in time.
Variational autoencoder for deep learning of images, labels and captions
Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin · 2016
Closest in time.
Adaptive feature abstraction for translating video to language
Original
Y. Pu, M. R. Min, Z. Gan, and L. Carin · 2016
Closest in time.
Factored temporal sigmoid belief networks for sequence learning
J. Song, Z. Gan, and L. Carin · 2016
Closest in time.
Theano: A Python framework for fast computation of mathematical expressions
Theano Development Team · 2016
Closest in time.
Rich image captioning in the wild
K. Tran, X. He, L. Zhang, J. Sun, C. Carapcea, C. Thrasher, C. Buehler, and C. Sienkiewicz · 2016
Closest in time.
What value do explicit high level concepts have in vision to language problems?
Q. Wu, C. Shen, L. Liu, A. Dick, and A. v. d. Hengel · 2016
Closest in time.
On multiplicative integration with recurrent neural networks
Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. Salakhutdinov · 2016
Closest in time.
Msr-vtt: A large video description dataset for bridging video and language
J. Xu, T. Mei, T. Yao, and Y. Rui · 2016
Closest in time.
Review networks for caption generation
Z. Yang, Y. Yuan, Y. Wu, R. Salakhutdinov, and W. W. Cohen · 2016
Closest in time.
Image captioning with semantic attention
Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo · 2016
Closest in time.
Video paragraph captioning using hierarchical recurrent neural networks
H. Yu, J. Wang, Z. Huang, Y. Yang, and W. Xu · 2016
Closest in time.
Stylenet: Generating attractive visual captions with styles
C. Gan, Z. Gan, X. He, J. Gao, and L. Deng · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…