2019

Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

Hu, Hexiang, Misra, Ishan, van der Maaten, Laurens

Understand

Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision.

Tasks such as text-based image retrieval and image captioning were designed to test this ability but come with evaluation measures that have a high variance or are difficult to interpret.
We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images.
The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates the reliability problems of retrieval evaluations, and focuses on the system's ability to understand fine-grained visual structure.

Built on

Computational interpretations of the gricean maxims in the generation of referring expressions
R. Dale and E. Reiter · 1995
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour · 2000
Earlier work this paper cites.
Learning the semantics of words and pictures
K. Barnard and D. Forsyth · 2001
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu · 2002
Earlier work this paper cites.
Matching words and pictures
K. Barnard, P. Duygulu, D. Forsyth, N. d. Freitas, D. M. Blei, and M. I. Jordan · 2003
Earlier work this paper cites.
Wordnet:: Similarity: measuring the relatedness of concepts
T. Pedersen, S. Patwardhan, and J. Michelizzi · 2004
Earlier work this paper cites.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments
S. Banerjee and A. Lavie · 2005
Earlier work this paper cites.
Understanding and predicting importance in images
A. C. Berg, T. L. Berg, H. Daume, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, et al · 2012
Earlier work this paper cites.
Midge: Generating image descriptions from computer vision detections
M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III · 2012
Earlier work this paper cites.
Label-embedding for attribute-based classification
Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid · 2013
Earlier work this paper cites.
Framing image description as a ranking task: Data, models and evaluation metrics
M. Hodosh, P. Young, and J. Hockenmaier · 2013
Earlier work this paper cites.
Babytalk: Understanding and generating simple image descriptions
G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg · 2013
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio · 2014
Earlier work this paper cites.
A multi-view embedding space for modeling internet images, tags, and their semantics
Y. Gong, Q. Ke, M. Isard, and S. Lazebnik · 2014
Earlier work this paper cites.
Referitgame: Referring to objects in photographs of natural scenes
S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg · 2014
Earlier work this paper cites.
Unifying visual-semantic embeddings with multimodal neural language models
R. Kiros, R. Salakhutdinov, and R. S. Zemel · 2014
Earlier work this paper cites.
What are you talking about? text-to-image coreference
C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler · 2014
Earlier work this paper cites.
Grounded compositional semantics for finding and describing images with sentences
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier · 2014
Earlier work this paper cites.

Similar

Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick · 2015
Cited alongside, same era.
Mind’s eye: A recurrent visual representation for image caption generation
X. Chen and C. Lawrence Zitnick · 2015
Cited alongside, same era.
From captions to visual concepts and back
H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, et al · 2015
Cited alongside, same era.
A survey of current datasets for vision and language research
Original
F. Ferraro, N. Mostafazadeh, L. Vanderwende, J. Devlin, M. Galley, M. Mitchell, et al · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba · 2015
Cited alongside, same era.
Ask your neurons: A neural-based approach to answering questions about images
M. Malinowski, M. Rohrbach, and M. Fritz · 2015
Cited alongside, same era.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models
B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
An embarrassingly simple approach to zero-shot learning
B. Romera-Paredes and P. Torr · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Cited alongside, same era.

Then

Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels
I. Misra, C. Lawrence Zitnick, M. Mitchell, and R. Girshick · 2016
Later among the works it cites.
Generating natural questions about an image
N. Mostafazadeh, I. Misra, J. Devlin, M. Mitchell, X. He, and L. Vanderwende · 2016
Later among the works it cites.
Modeling context in referring expressions
L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg · 2016
Later among the works it cites.
Learning to ask: Neural question generation for reading comprehension
Original
X. Du, J. Shao, and C. Cardie · 2017
Later among the works it cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2017
Later among the works it cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al · 2017
Later among the works it cites.
Learning visual n-grams from web data
A. Li, A. Jabri, A. Joulin, and L. van der Maaten · 2017
Later among the works it cites.
Self-critical sequence training for image captioning
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel · 2017
Later among the works it cites.
Context-aware captions from context-agnostic supervision
R. Vedantam, S. Bengio, K. Murphy, D. Parikh, and G. Chechik · 2017
Later among the works it cites.
Aggregated residual transformations for deep neural networks
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2017
Later among the works it cites.
http://lsun.cs.princeton.edu/slides/caption_open.pdf
Microsoft COCO 1st Captioning Challenge (Large-scale Scene UNderstanding Workshop, CVPR 2015) · 2018
Later among the works it cites.
Bottom-up and top-down attention for image captioning and visual question answering
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2018
Later among the works it cites.
Convolutional image captioning
J. Aneja, A. Deshpande, and A. G. Schwing · 2018
Later among the works it cites.
Vse++: Improving visual-semantic embeddings with hard negatives
F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler · 2018
Later among the works it cites.
Stacked cross attention for image-text matching
K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He · 2018
Later among the works it cites.
Discriminability objective for training descriptive captions
R. Luo, B. Price, S. Cohen, and G. Shakhnarovich · 2018
Later among the works it cites.
Exploring the limits of weakly supervised pretraining
D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten · 2018
Later among the works it cites.
Learning by asking questions
I. Misra, R. Girshick, R. Fergus, M. Hebert, A. Gupta, and L. van der Maaten · 2018
Later among the works it cites.
Engaging image captioning via personality
Original
K. Shuster, S. Humeau, H. Hu, A. Bordes, and J. Weston · 2018
Later among the works it cites.
A corpus for reasoning about natural language grounded in photographs
A. Suhr, S. Zhou, I. Zhang, H. Bai, and Y. Artzi · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…