2016

GuessWhat?! Visual object discovery through multi-modal dialogue

de Vries, Harm, Strub, Florian, Chandar, Sarath et al.

Understand

We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems.

The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions.
Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task.
Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images.

Built on

Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
A stochastic model of computer-human interaction for learning dialogue strategies
E. Levin and R. Pieraccini · 1997
Earlier work this paper cites.
Reinforcement Learning for Spoken Dialogue Systems
S. Singh, M. Kearns, D. Litman, and M. Walker · 1999
Earlier work this paper cites.
Labeling images with a computer game
L. V. Ahn and L. Dabbish · 2004
Earlier work this paper cites.
Peekaboom: a game for locating objects in images
L. V. Ahn, R. Liu, and M. Blum · 2006
Earlier work this paper cites.
Dynamic topic models
D. Blei and J. Lafferty · 2006
Earlier work this paper cites.
One-shot learning of object categories
L. Fei-Fei, R. Fergus, and P. Perona · 2006
Earlier work this paper cites.
A probabilistic framework for dialog simulation and optimal strategy learning
O. Pietquin and T. Dutoit · 2006
Earlier work this paper cites.
A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies
J. Schatzmann, K. Weilhammer, M. Stuttle, and S. Young · 2006
Earlier work this paper cites.
Natural language processing with Python
S. Bird, E. Klein, and E. Loper · 2009
Earlier work this paper cites.
The segmented and annotated IAPR TC-12 benchmark
H. Escalante, C. Hernández, J. Gonzalez, A. López-López, M. Montes, E. Morales, E. Sucar, L. Villaseñ, and M. Grubinger · 2010
Earlier work this paper cites.
Software Framework for Topic Modelling with Large Corpora
R. Řehůřek and P. Sojka · 2010
Earlier work this paper cites.
Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data?
M. Buhrmester, T. Kwang, and S. Gosling · 2011
Earlier work this paper cites.
Computational generation of referring expressions: A survey
E. Krahmer and K. V. Deemter · 2012
Earlier work this paper cites.
Data-Driven Methods for Adaptive Spoken Dialogue Systems
O. Lemon and O. Pietquin, editor · 2012
Earlier work this paper cites.
A survey on metrics for the evaluation of user simulations
O. Pietquin and H. Hastie · 2013
Earlier work this paper cites.
POMDP-based statistical spoken dialog systems: A review
S. Young, M. Gašić, B. Thomson, and J. Williams · 2013
Earlier work this paper cites.

Similar

Learning phrase representations using RNN encoder-decoder for statistical machine translation
K. Cho, B. V. Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Cited alongside, same era.
ReferItGame: Referring to Objects in Photographs of Natural Scenes
S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg · 2014
Cited alongside, same era.
Adam: A Method for Stochastic Optimization
Original
D. P. Kingma and J. Ba · 2014
Cited alongside, same era.
Microsoft coco: Common objects in context
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick · 2014
Cited alongside, same era.
A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. Le · 2014
Cited alongside, same era.
Learning deep features for scene recognition using places database
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva · 2014
Cited alongside, same era.
Vqa: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, Z. Lawrence, and D. Parikh · 2015
Cited alongside, same era.
Visual turing test for computer vision systems
D. Geman, S. Geman, N. Hallonquist, and L. Younes · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Cited alongside, same era.
Deep learning
Y. LeCun, Y. Bengio, and G. Hinton · 2015
Cited alongside, same era.
Generation and comprehension of unambiguous object descriptions
Original
J. Mao, J. Huang, A. Toshev, O. Camburu, A. Yuille, and K. Murphy · 2015
Cited alongside, same era.

Then

Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio · 2015
Later among the works it cites.
http://www.20q.net/
20 Questions · 2016
Closest in time.
en.akinator.com/
Akinator · 2016
Closest in time.
Analyzing the Behavior of Visual Question Answering Models
Original
A. Agrawal, D. Batra, and D. Parikh · 2016
Closest in time.
Learning Aligned Cross-Modal Representations from Weakly Aligned Data
L. Castrejon, Y. Aytar, C. Vondrick, H. Pirsiavash, and A. Torralba · 2016
Closest in time.
Evaluating prerequisite qualities for learning end-to-end dialog systems
J. Dodge, A. Gane, X. Zhang, A. Bordes, S. Chopra, A. Miller, A. Szlam, and J. Weston · 2016
Closest in time.
Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Closest in time.
Natural Language Object Retrieval
R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell · 2016
Closest in time.
Revisiting Visual Question Answering Baselines
A. Jabri, A. Joulin, and L. van der Maaten · 2016
Closest in time.
How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation
Original
C. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, and J. Pineau · 2016
Closest in time.
Hierarchical Question-Image Co-Attention for Visual Question Answering
Original
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Closest in time.
Where to look: Focus regions for visual question answering
K. Shih, S. Singh, and D. Hoiem · 2016
Closest in time.
A Network-based End-to-End Trainable Task-oriented Dialogue System
Original
T. Wen, M. Gasic, N. Mrksic, L. Rojas-Barahona, P. Su, S. Ultes, D. Vandyke, and S. Young · 2016
Closest in time.
Towards ai-complete question answering: A set of prerequisite toy tasks
J. Weston, A. Bordes, S. Chopra, A. Rush, B. van Merriënboer, A. Joulin, and T. Mikolov · 2016
Closest in time.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola · 2016
Closest in time.
Modeling context in referring expressions
L. Yu, P. Poirson, S. Yang, A. Berg, and T. Berg · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…