2014

Towards a Visual Turing Challenge

Malinowski, Mateusz, Fritz, Mario

Understand

As language and visual understanding by machines progresses rapidly, we are observing an increasing interest in holistic architectures that tightly interlink both modalities in a joint learning and inference process.

This trend has allowed the community to progress towards more challenging and open tasks and refueled the hope at achieving the old AI dream of building machines that could pass a turing test in open domains.
In order to steadily make progress towards this goal, we realize that quantifying performance becomes increasingly difficult.
Therefore we ask how we can precisely define such challenges and how we can evaluate different algorithms on this open tasks? In this paper, we summarize and discuss such challenges as well as try to give answers where appropriate options are available in the literature.

Built on

Computing machinery and intelligence
Turing, A.M.: · 1950
Earlier work this paper cites.
Natural categories
Rosch, E.H.: · 1973
Earlier work this paper cites.
Women, fire, and dangerous things: What categories reveal about the mind
Lakoff, G.: · 1990
Earlier work this paper cites.
The symbol grounding problem
Harnad, S.: · 1990
Earlier work this paper cites.
Verbs semantics and lexical selection
Wu, Z., Palmer, M.: · 1994
Earlier work this paper cites.
Wordnet: a lexical database for english
Miller, G.A.: · 1995
Earlier work this paper cites.
Learning to parse database queries using inductive logic programming
Zelle, J.M., Mooney, R.J.: · 1996
Earlier work this paper cites.
Space in language and cognition: Explorations in cognitive diversity. Volume 5
Levinson, S.C.: · 2003
Earlier work this paper cites.
Feature-rich part-of-speech tagging with a cyclic dependency network
Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: · 2003
Earlier work this paper cites.
Representation and Inference for Natural Language. A First Course in Computational Semantics
Blackburn, P., Bos, J.: · 2005
Earlier work this paper cites.
Online learning of relaxed ccg grammars for parsing to logical form
Zettlemoyer, L.S., Collins, M.: · 2007
Earlier work this paper cites.
Inducing probabilistic ccg grammars from logical form with higher-order unification
Kwiatkowski, T., Zettlemoyer, L., Goldwater, S., Steedman, M.: · 2010
Earlier work this paper cites.
Every picture tells a story: Generating sentences from images
Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: · 2010
Earlier work this paper cites.
Online crowdsourcing: rating annotators and obtaining cost-effective labels
Welinder, P., Perona, P.: · 2010
Earlier work this paper cites.
The multidimensional wisdom of crowds
Welinder, P., Branson, S., Perona, P., Belongie, S.J.: · 2010
Earlier work this paper cites.
Every picture tells a story: Generating sentences from images
Farhadi, A., Hejrati, M., Sadeghi, M.A., Young, P., Rashtchian, C., Hockenmaier, J., Forsyth, D.: · 2010
Earlier work this paper cites.
What does classifying more than 10,000 image categories tell us?
Deng, J., Berg, A.C., Li, K., Fei-Fei, L.: · 2010
Earlier work this paper cites.
A game-theoretic approach to generating spatial descriptions
Golland, D., Liang, P., Klein, D.: · 2010
Earlier work this paper cites.
Collecting image annotations using amazon’s mechanical turk
Rashtchian, C., Young, P., Hodosh, M., Hockenmaier, J.: · 2010
Earlier work this paper cites.
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: · 2011
Earlier work this paper cites.
Criticisms of the turing test and why you should ignore (most of) them
LaCurts, K.: · 2011
Earlier work this paper cites.
Contour detection and hierarchical image segmentation
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: · 2011
Earlier work this paper cites.

Similar

Wsabie: Scaling up to large vocabulary image annotation
Weston, J., Bengio, S., Usunier, N.: · 2011
Cited alongside, same era.
Evaluating knowledge transfer and zero-shot learning in a large-scale setting
Rohrbach, M., Stark, M., Schiele, B.: · 2011
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., Hinton, G.E.: · 2012
Cited alongside, same era.
A joint model of language and perception for grounded attribute learning
Matuszek, C., Fitzgerald, N., Zettlemoyer, L., Bo, L., Fox, D.: · 2012
Cited alongside, same era.
Towards good practice in large-scale learning for image classification
Perronnin, F., Akata, Z., Harchaoui, Z., Schmid, C.: · 2012
Cited alongside, same era.
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: · 2012
Cited alongside, same era.
Image retrieval with structured object queries using latent ranking svm
Lan, T., Yang, W., Wang, Y., Mori, G.: · 2012
Cited alongside, same era.
Script data for attribute-based recognition of composite activities
Rohrbach, M., Regneri, M., Andriluka, M., Amin, S., Pinkal, M., Schiele, B.: · 2012
Cited alongside, same era.
Strong appearance and expressive spatial models for human pose estimation
Pishchulin, L., Andriluka, M., Gehler, P., Schiele, B.: · 2013
Cited alongside, same era.
Distributed representations of words and phrases and their compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: · 2013
Cited alongside, same era.
Jointly learning to parse and perceive: Connecting natural language to the physical world
Krishnamurthy, J., Kollar, T.: · 2013
Cited alongside, same era.
Learning dependency-based compositional semantics
Liang, P., Jordan, M.I., Klein, D.: · 2013
Cited alongside, same era.

Then

Rich feature hierarchies for accurate object detection and semantic segmentation
Girshick, R., Donahue, J., Darrell, T., Malik, J.: · 2014
Closest in time.
Joint training of a convolutional network and a graphical model for human pose estimation
Tompson, J., Jain, A., LeCun, Y., Bregler, C.: · 2014
Closest in time.
Spatial pyramid pooling in deep convolutional networks for visual recognition
He, K., Zhang, X., Ren, S., Sun, J.: · 2014
Closest in time.
Deeply-supervised nets
Original
Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: · 2014
Closest in time.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K., Zisserman, A.: · 2014
Closest in time.
Learning phrase representations using rnn encoder-decoder for statistical machine translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., Bahdanau, D., Bengio, Y.: · 2014
Closest in time.
Coherent multi-sentence video description with variable level of detail
Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Amin, S., Andriluka, M., Pinkal, M., Schiele, B.: · 2014
Closest in time.
Grounded compositional semantics for finding and describing images with sentences
Socher, R., Karpathy, A., Le, Q., Manning, C., Ng, A.: · 2014
Closest in time.
Deep fragment embeddings for bidirectional image sentence mapping
Karpathy, A., Joulin, A., Fei-Fei, L.: · 2014
Closest in time.
Explain images with multimodal recurrent neural networks
Original
Mao, J., Xu, W., Yang, Y., Wang, J., Yuille, A.L.: · 2014
Closest in time.
What are you talking about? text-to-image coreference
Kong, C., Lin, D., Bansal, M., Urtasun, R., Fidler, S.: · 2014
Closest in time.
Semantic parsing via paraphrasing
Berant, J., Liang, P.: · 2014
Closest in time.
A neural network for factoid question answering over paragraphs
Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., III, H.D.: · 2014
Closest in time.
Open question answering over curated and extracted knowledge bases
Fader, A., Zettlemoyer, L., Etzioni, O.: · 2014
Closest in time.
A multi-world approach to question answering about real-world scenes based on uncertain input
Malinowski, M., Fritz, M.: · 2014
Closest in time.
LSDA: Large scale detection through adaptation
Hoffman, J., Guadarrama, S., Tzeng, E., Hu, R., Donahue, J., Girshick, R., Darrell, T., Saenko, K.: · 2014
Closest in time.
Low-dimensional embeddings of logic
Rocktäschel, T., Bosnjak, M., Singh, S., Riedel, S.: · 2014
Closest in time.
Combining formal and distributional models of temporal and intensional semantics
Lewis, M., Steedman, M.: · 2014
Closest in time.
Imagenet large scale visual recognition challenge
Original
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: · 2014
Closest in time.
Improving image-sentence embeddings using large weakly annotated photo collections
Gong, Y., Wang, L., Hodosh, M., Hockenmaier, J., Lazebnik, S.: · 2014
Closest in time.
Glove: Global vectors for word representation
Pennington, J., Socher, R., Manning, C.D.: · 2014
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…