2015

Order-Embeddings of Images and Language

Vendrov, Ivan, Kiros, Ryan, Fidler, Sanja et al.

Understand

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images.

In this paper we advocate for explicitly modeling the partial order structure of this hierarchy.
Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involving images and language.
We show that the resulting representations improve performance over current approaches for hypernym prediction and image-caption retrieval.

Built on

Wordnet: a lexical database for english
Miller, George A · 1995
Earlier work this paper cites.
Learning a similarity metric discriminatively, with application to face verification
Chopra, Sumit, Hadsell, Raia, and LeCun, Yann · 2005
Earlier work this paper cites.
Visualizing data using t-sne
Van der Maaten, Laurens and Hinton, Geoffrey · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li · 2009
Earlier work this paper cites.
Learning structured embeddings of knowledge bases
Bordes, Antoine, Weston, Jason, Collobert, Ronan, and Bengio, Yoshua · 2011
Earlier work this paper cites.
Entailment above the word level in distributional semantics
Baroni, Marco, Bernardi, Raffaella, Do, Ngoc-Quynh, and Shan, Chung-chieh · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E · 2012
Earlier work this paper cites.
Devise: A deep visual-semantic embedding model
Frome, Andrea, Corrado, Greg S, Shlens, Jon, Bengio, Samy, Dean, Jeff, Mikolov, Tomas, et al · 2013
Earlier work this paper cites.
Framing image description as a ranking task: Data, models and evaluation metrics
Hodosh, Micah, Young, Peter, and Hockenmaier, Julia · 2013
Earlier work this paper cites.
Linguistic regularities in continuous space word representations
Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey · 2013
Earlier work this paper cites.

Similar

Reasoning with neural tensor networks for knowledge base completion
Socher, Richard, Chen, Danqi, Manning, Christopher D, and Ng, Andrew · 2013
Cited alongside, same era.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Young, Peter, Lai, Alice, Hodosh, Micah, and Hockenmaier, Julia · 2013
Cited alongside, same era.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua · 2014
Cited alongside, same era.
Unifying visual-semantic embeddings with multimodal neural language models
Original
Kiros, Ryan, Salakhutdinov, Ruslan, and Zemel, Richard S · 2014
Cited alongside, same era.
Zero-shot learning by convex combination of semantic embeddings
Norouzi, Mohammad, Mikolov, Tomas, Bengio, Samy, Singer, Yoram, Shlens, Jonathon, Frome, Andrea, Corrado, Greg S, and Dean, Jeffrey · 2014
Cited alongside, same era.
Grounded compositional semantics for finding and describing images with sentences
Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y · 2014
Cited alongside, same era.
A large annotated corpus for learning natural language inference
Bowman, Samuel R., Angeli, Gabor, Potts, Christopher, and Manning, Christopher D · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, Sergey and Szegedy, Christian · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
Karpathy, Andrej and Li, Fei-Fei · 2015
Cited alongside, same era.
Visual semantic search: Retrieving videos via complex textual queries
Lin, Dahua, Fidler, Sanja, Kong, Chen, and Urtasun, Raquel
Cited in the paper.
Microsoft coco: Common objects in context
Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Hays, James, Perona, Pietro, Ramanan, Deva, Dollár, Piotr, and Zitnick, C Lawrence
Cited in the paper.

Then

Adam: A method for stochastic optimization
Kingma, Diederik and Ba, Jimmy · 2015
Closest in time.
Skip-thought vectors
Kiros, Ryan, Zhu, Yukun, Salakhutdinov, Ruslan, Zemel, Richard S, Torralba, Antonio, Urtasun, Raquel, and Fidler, Sanja · 2015
Closest in time.
Associating neural word embeddings with deep image representations using fisher vectors
Klein, Benjamin, Lev, Guy, Sadeh, Gil, and Wolf, Lior · 2015
Closest in time.
Multimodal convolutional neural networks for matching image and sentence
Ma, Lin, Lu, Zhengdong, Shang, Lifeng, and Li, Hang · 2015
Closest in time.
Deep captioning with multimodal recurrent neural networks (m-rnn)
Mao, Junhua, Xu, Wei, Yang, Yi, Wang, Jiang, and Yuille, Alan · 2015
Closest in time.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
Original
Plummer, Bryan, Wang, Liwei, Cervantes, Chris, Caicedo, Juan, Hockenmaier, Julia, and Lazebnik, Svetlana · 2015
Closest in time.
Reasoning about entailment with neural attention
Original
Rocktäschel, Tim, Grefenstette, Edward, Hermann, Karl Moritz, Kočiskỳ, Tomáš, and Blunsom, Phil · 2015
Closest in time.
Very deep convolutional networks for large-scale image recognition
Simonyan, K. and Zisserman, A · 2015
Closest in time.
Word representations via gaussian embedding
Vilnis, Luke and McCallum, Andrew · 2015
Closest in time.
Show and tell: A neural image caption generator
Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…