2020

Captioning Images Taken by People Who Are Blind

Gurari, Danna, Zhao, Yinan, Zhang, Meng et al.

Understand

While an important problem in the vision community is to design algorithms that can automatically caption images, few publicly-available datasets for algorithm development directly address the interests of real users.

Observing that people who are blind have relied on (human-based) image captioning services to learn about images they take for nearly a decade, we introduce the first image captioning dataset to represent this real use case.
This new dataset, which we call VizWiz-Captions, consists of over 39,000 images originating from people who are blind that are each paired with five captions.
We analyze this dataset to (1) characterize the typical captions, (2) characterize the diversity of content found in the images, and (3) compare its content to that found in eight popular vision datasets.

Built on

BLEU: A method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu · 2002
Earlier work this paper cites.
Rouge: A package for automatic evaluation of summaries
C.-Y. Lin · 2004
Earlier work this paper cites.
Describing images on the web: A survey of current practice and prospects for the future
H. Petrie, C. Harrison, and S. Dev · 2005
Earlier work this paper cites.
The iapr tc-12 benchmark: A new evaluation resource for visual information systems
M. Grubinger, P. Clough, H. Müller, and T. Deselaers · 2006
Earlier work this paper cites.
Improving accessibility of the web with a computer game
L. Von Ahn, S. Ginosar, M. Kedia, R. Liu, and M. Blum · 2006
Earlier work this paper cites.
Automatic image annotation using auxiliary text information
Y. Feng and M. Lapata · 2008
Earlier work this paper cites.
VizWiz: Nearly real-time answers to visual questions
J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, and S. White · 2010
Earlier work this paper cites.
Every picture tells a story: Generating sentences from images
A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth · 2010
Earlier work this paper cites.
Collecting image annotations using Amazon’s Mechanical Turk
C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier · 2010
Earlier work this paper cites.
Sun database: Large-scale scene recognition from abbey to zoo
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba · 2010
Earlier work this paper cites.
Crowdsourcing subjective fashion advice using VizWiz: Challenges and opportunities
M. A. Burton, E. Brady, R. Brewer, C. Neylan, J. P. Bigham, and A. Hurst · 2012
Earlier work this paper cites.
Sun attribute database: Discovering, annotating, and recognizing scene attributes
G. Patterson and J. Hays · 2012
Earlier work this paper cites.
Visual challenges in the everyday lives of blind people
E. Brady, M. R. Morris, Y. Zhong, S. White, and J. P. Bigham · 2013
Earlier work this paper cites.
Image description using visual dependency representations
D. Elliott and F. Keller · 2013
Earlier work this paper cites.
Framing image description as a ranking task: Data, models and evaluation metrics
M. Hodosh, P. Young, and J. Hockenmaier · 2013
Earlier work this paper cites.
Babytalk: Understanding and generating simple image descriptions
G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg · 2013
Earlier work this paper cites.
Learning the visual interpretation of sentences
C. L. Zitnick, D. Parikh, and L. Vanderwende · 2013
Earlier work this paper cites.
Meteor universal: Language specific translation evaluation for any target language
M. Denkowski and A. Lavie · 2014
Earlier work this paper cites.
What are you talking about? text-to-image coreference
C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler · 2014
Earlier work this paper cites.

Similar

Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
P. Young, A. Lai, M. Hodosh, and J. Hockenmaier · 2014
Cited alongside, same era.
Learning deep features for scene recognition using places database
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva · 2014
Cited alongside, same era.
Déja image-captions: A corpus of expressive descriptions in repetition
J. Chen, P. Kuznetsova, D. Warren, and Y. Choi · 2015
Cited alongside, same era.
Microsoft COCO captions: Data collection and evaluation server
Original
X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick · 2015
Cited alongside, same era.
Deep multimodal semantic embeddings for speech and images
D. Harwath and J. Glass · 2015
Cited alongside, same era.
Image specificity
M. Jas and D. Parikh · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, and M. Bernstein · 2015
Cited alongside, same era.
Cider: Consensus-based image description evaluation
R. Vedantam, C. Lawrence Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Regionspeak: Quick comprehensive spatial descriptions of complex images for blind users
Y. Zhong, W. S. Lasecki, E. Brady, and J. P. Bigham · 2015
Cited alongside, same era.
Spice: Semantic propositional image caption evaluation
P. Anderson, B. Fernando, M. Johnson, and S. Gould · 2016
Cited alongside, same era.

Then

Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Network Service
S. Wu, J. Wieland, O. Farivar, and J. Schiller · 2017
Later among the works it cites.
Stair captions: Constructing a large-scale japanese image caption dataset
Original
Y. Yoshikawa, Y. Shigeto, and A. Takeuchi · 2017
Later among the works it cites.
Nocaps: Novel object captioning at scale
Original
H. Agrawal, K. Desai, X. Chen, R. Jain, D. Batra, D. Parikh, S. Lee, and P. Anderson · 2018
Later among the works it cites.
Bottom-up and top-down attention for image captioning and visual question answering
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang · 2018
Later among the works it cites.
A survey on automatic image caption generation
S. Bai and S. An · 2018
Later among the works it cites.
How Teens with Visual Impairments Take, Edit, and Share Photos on Social Media
C. L. Bennett, M. E. Mott, E. Cutrell, and M. R. Morris · 2018
Later among the works it cites.
Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search
D. Guinness, E. Cutrell, and M. R. Morris · 2018
Later among the works it cites.
VizWiz Grand Challenge: Answering Visual Questions from Blind People
D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham · 2018
Later among the works it cites.
Rich Representations of Visual Content for Screen Reader Users
M. R. Morris, J. Johnson, C. L. Bennett, and E. Cutrell · 2018
Later among the works it cites.
Evaluating and Complementing Vision-to-Language Technology for People who are Blind with Conversational Crowdsourcing
E. Salisbury, E. Kamar, and M. R. Morris · 2018
Later among the works it cites.
Engaging image captioning via personality
Original
K. Shuster, S. Humeau, H. Hu, A. Bordes, and J. Weston · 2018
Later among the works it cites.
A survey on automatic image captioning
G. Srivastava and R. Srivastava · 2018
Later among the works it cites.
VizWiz-Priv: A Dataset for Recognizing the Presence and Purpose of Private Visual Information in Images Taken by Blind People
D. Gurari, Q. Li, C. Lin, Y. Zhao, A. Guo, A. Stangl, and J. P. Bigham · 2019
Later among the works it cites.
A comprehensive survey of deep learning for image captioning
M. D. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga · 2019
Later among the works it cites.
Attention on attention for image captioning
L. Huang, W. Wang, J. Chen, and X.-Y. Wei · 2019
Later among the works it cites.
Auto-encoding scene graphs for image captioning
X. Yang, K. Tang, H. Zhang, and J. Cai · 2019
Later among the works it cites.
Assessing image quality issues for real-world problems
T.-Y. Chiu, Y. Zhao, and D. Gurari · 2020
Closest in time.
” person, shoes, tree. is the person naked?” what people with vision impairments want in image descriptions
A. Stangl, M. R. Morris, and D. Gurari · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…