2017

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Anderson, Peter, He, Xiaodong, Buehler, Chris et al.

Understand

Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning.

In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions.
This is the natural basis for attention to be considered.
Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings.

Built on

A feature-integration theory of attention
A. M. Treisman and G. Gelade · 1980
Earlier work this paper cites.
Perceptual grouping and attention in visual search for features and for objects
A. Treisman · 1982
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
R. J. Williams · 1992
Earlier work this paper cites.
Shifting visual attention between objects and locations: evidence from normal and parietal lesion subjects
R. Egly, J. Driver, and R. D. Rafal · 1994
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Objects and attention: The state of the art
B. J. Scholl · 2001
Earlier work this paper cites.
Control of goal-directed and stimulus-driven attention in the brain
M. Corbetta and G. L. Shulman · 2002
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
K. Papineni, S. Roukos, T. Ward, and W. Zhu · 2002
Earlier work this paper cites.
Rouge: a package for automatic evaluation of summaries
C. Lin · 2004
Earlier work this paper cites.
Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices
T. J. Buschman and E. K. Miller · 2007
Earlier work this paper cites.
ADADELTA: an adaptive learning rate method
Original
M. D. Zeiler · 2012
Earlier work this paper cites.
Selective search for object recognition
J. R. Uijlings, K. E. Van De Sande, T. Gevers, and A. W. Smeulders · 2013
Earlier work this paper cites.
Learning phrase representations using RNN encoder-decoder for statistical machine translation
K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio · 2014
Earlier work this paper cites.
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
M. Denkowski and A. Lavie · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick · 2014
Earlier work this paper cites.
GloVe: Global Vectors for Word Representation
J. Pennington, R. Socher, and C. D. Manning · 2014
Earlier work this paper cites.
Edge boxes: Locating object proposals from edges
L. Zitnick and P. Dollár · 2014
Earlier work this paper cites.
VQA: Visual Question Answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh · 2015
Earlier work this paper cites.

Similar

Microsoft COCO captions: Data collection and evaluation server
Original
X. Chen, T.-Y. L. Hao Fang, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick · 2015
Cited alongside, same era.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Cited alongside, same era.
Spatial transformer networks
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu · 2015
Cited alongside, same era.
Aligning where to see and what to tell: image caption with region-based attention and scene factorization
Original
J. Jin, K. Fu, R. Cui, F. Sha, and C. Zhang · 2015
Cited alongside, same era.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and F.-F. Li · 2015
Cited alongside, same era.
SSD: Single shot multibox detector
Original
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg · 2015
Cited alongside, same era.
Deeper lstm and normalized cnn visual question answering model
J. Lu, X. Lin, D. Batra, and D. Parikh · 2015
Cited alongside, same era.
Faster R-CNN: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Cited alongside, same era.
Highway networks
Original
R. K. Srivastava, K. Greff, and J. Schmidhuber · 2015
Cited alongside, same era.
CIDEr: Consensus-based image description evaluation
R. Vedantam, C. L. Zitnick, and D. Parikh · 2015
Cited alongside, same era.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio · 2015
Cited alongside, same era.

Then

Revisiting visual question answering baselines
Original
A. Jabri, A. Joulin, and L. van der Maaten · 2016
Later among the works it cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Hierarchical question-image co-attention for visual question answering
J. Lu, J. Yang, D. Batra, and D. Parikh · 2016
Later among the works it cites.
You only look once: Unified, real-time object detection
J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi · 2016
Later among the works it cites.
Zero-shot visual question answering
Original
D. Teney and A. van den Hengel · 2016
Later among the works it cites.
Ask, attend and answer: Exploring question-guided spatial attention for visual question answering
H. Xu and K. Saenko · 2016
Later among the works it cites.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. J. Smola · 2016
Later among the works it cites.
Review networks for caption generation
Z. Yang, Y. Yuan, Y. Wu, R. Salakhutdinov, and W. W. Cohen · 2016
Later among the works it cites.
Visual7w: Grounded question answering in images
Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh · 2017
Closest in time.
Show, ask, attend, and answer: A strong baseline for visual question answering
Original
V. Kazemi and A. Elqursh · 2017
Closest in time.
Improved image captioning via policy gradient optimization of spider
S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy · 2017
Closest in time.
Knowing when to look: Adaptive attention via a visual sentinel for image captioning
J. Lu, C. Xiong, D. Parikh, and R. Socher · 2017
Closest in time.
Areas of attention for image captioning
M. Pedersoli, T. Lucas, C. Schmid, and J. Verbeek · 2017
Closest in time.
Self-critical sequence training for image captioning
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel · 2017
Closest in time.
Boosting image captioning with attributes
T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei · 2017
Closest in time.
Tips and tricks for visual question answering: Learnings from the 2017 challenge
D. Teney, P. Anderson, X. He, and A. van den Hengel · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…