2018

VizWiz Grand Challenge: Answering Visual Questions from Blind People

Gurari, Danna, Li, Qing, Stangl, Abigale J. et al.

Understand

The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings.

We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting.
VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question.
VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered.

Built on

BLEU: A method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W. Zhu · 2002
Earlier work this paper cites.
ImageNet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Vizwiz: Nearly real-time answers to visual questions
J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, and T. Yeh · 2010
Earlier work this paper cites.
Vizwiz:: Locateit-enabling blind people to locate objects in their environment
J. P. Bigham, C. Jayant, A. Miller, B. White, and T. Yeh · 2010
Earlier work this paper cites.
Sun database: Large-scale scene recognition from abbey to zoo
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba · 2010
Earlier work this paper cites.
Supporting blind photography
C. Jayant, H. Ji, S. White, and J. P. Bigham · 2011
Earlier work this paper cites.
Crowdsourcing subjective fashion advice using VizWiz: Challenges and opportunities
M. A. Burton, E. Brady, R. Brewer, C. Neylan, J. P. Bigham, and A. Hurst · 2012
Earlier work this paper cites.
Sun attribute database: Discovering, annotating, and recognizing scene attributes
G. Patterson and J. Hays · 2012
Earlier work this paper cites.
Indoor segmentation and support inference from RGBD images
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus · 2012
Earlier work this paper cites.
A qualitative study to support a blind photography mobile application
D. Adams, L. Morales, and S. Kurniawan · 2013
Earlier work this paper cites.
Visual challenges in the everyday lives of blind people
E. Brady, M. R. Morris, Y. Zhong, S. White, and J. P. Bigham · 2013
Earlier work this paper cites.
Image description using visual dependency representations
D. Elliott and F. Keller · 2013
Earlier work this paper cites.
Answering visual questions with conversational crowd assistants
W. S. Lasecki, P. Thiha, Y. Zhong, E. Brady, and J. P. Bigham · 2013
Earlier work this paper cites.
Real time object scanning using a mobile phone and cloud-based visual search engine
Y. Zhong, P. J. Garrigues, and J. P. Bigham · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
D. Kingma and J. Ba · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick · 2014
Earlier work this paper cites.

Similar

A multi-world approach to question answering about real-world scenes based on uncertain input
M. Malinowski and M. Fritz · 2014
Cited alongside, same era.
Dropout: A simple way to prevent neural networks from overfitting
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov · 2014
Cited alongside, same era.
An assisted photography framework to help visually impaired users properly aim a camera
M. Vázquez and A. Steinfeld · 2014
Cited alongside, same era.
Privacy concerns and behaviors of people with visual impairments
T. Ahmed, R. Hoyle, K. Connelly, D. Crandall, and A. Kapadia · 2015
Cited alongside, same era.
VQA: Visual Question Answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T. Lin, R. Vedantam, S. K. Gupta, P. Dollár, and C. L. Zitnick · 2015
Cited alongside, same era.
Are you talking to a machine? dataset and methods for multilingual image question answering
Original
H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Cited alongside, same era.
Exploring models and data for image question answering
M. Ren, R. Kiros, and R. S. Zemel · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Cited alongside, same era.
CIDEr: Consensus-based image description evaluation
R. Vedantam, L. C. Zitnick, and D. Parikh · 2015
Cited alongside, same era.

Then

Making the V in VQA matter: Elevating the role of image understanding in visual question answering
Original
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2016
Later among the works it cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning
Original
J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick · 2016
Later among the works it cites.
Question relevance in VQA: Identifying non-visual and false-premise questions
Original
A. Ray, G. Christie, M. Bansal, D. Batra, and D. Parikh · 2016
Later among the works it cites.
YFCC100M: The new data in multimedia research
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L. Li · 2016
Later among the works it cites.
Visual7W: Grounded question answering in images
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Bottom-up and top-down attention for image captioning and visual question answering
Original
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2017
Later among the works it cites.
Visual dialog
A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra · 2017
Later among the works it cites.
An analysis of visual question answering algorithms
Original
K. Kafle and C. Kanan · 2017
Later among the works it cites.
Show, ask, attend, and answer: A strong baseline for visual question answering
Original
V. Kazemi and A. Elqursh · 2017
Later among the works it cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, et al · 2017
Later among the works it cites.
Understanding blind people’s experiences with computer-generated captions of social media images
H. MacLeod, C. L. Bennett, M. R. Morris, and E. Cutrell · 2017
Later among the works it cites.
The promise of premise: Harnessing question premises in visual question answering
Original
A. Mahendru, V. Prabhu, A. Mohapatra, D. Batra, and S. Lee · 2017
Later among the works it cites.
Question part relevance and editing for cooperative and context-aware vqa (c2vqa)
A. S. Toor, H. Wechsler, and M. Nappi · 2017
Later among the works it cites.
FVQA: fact-based visual question answering
P. Wang, Q. Wu, C. Shen, A. Dick, and A. Hengel · 2017
Later among the works it cites.
Automatic alt-text: Computer-generated image descriptions for blind users on a social network service
S. Wu, J. Wieland, O. Farivar, and J. Schiller · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…