2018

Iterative Visual Reasoning Beyond Convolutions

Chen, Xinlei, Li, Li-Jia, Fei-Fei, Li et al.

Understand

We present a novel framework for iterative visual reasoning.

Our framework goes beyond current recognition systems that lack the capability to reason beyond stack of convolutions.
The framework consists of two core modules: a local module that uses spatial memory to store previous beliefs with parallel updates; and a global graph-reasoning module.
Our graph module has three components: a) a knowledge graph where we represent classes as nodes and build edges to encode different types of semantic relationships between them; b) a region graph of the current image where regions in the image are nodes and spatial relationships between these regions are edges; c) an assignment graph that assigns regions to classes.

Built on

Physical symbol systems
A. Newell · 1980
Earlier work this paper cites.
Scene perception: Detecting and judging objects undergoing relational violations
I. Biederman, R. J. Mezzanotte, and J. C. Rabinowitz · 1982
Earlier work this paper cites.
Interpretation as abduction
J. R. Hobbs, M. Stickel, P. Martin, and D. Edwards · 1988
Earlier work this paper cites.
The symbol grounding problem
S. Harnad · 1990
Earlier work this paper cites.
Item-based collaborative filtering recommendation algorithms
B. Sarwar, G. Karypis, J. Konstan, and J. Riedl · 2001
Earlier work this paper cites.
Context-based vision system for place and object recognition
A. Torralba, K. P. Murphy, W. T. Freeman, M. A. Rubin, et al · 2003
Earlier work this paper cites.
One-shot learning of object categories
L. Fei-Fei, R. Fergus, and P. Perona · 2006
Earlier work this paper cites.
The link-prediction problem for social networks
D. Liben-Nowell and J. Kleinberg · 2007
Earlier work this paper cites.
An empirical study of context in object detection
S. K. Divvala, D. Hoiem, J. H. Hays, A. A. Efros, and M. Hebert · 2009
Earlier work this paper cites.
Describing objects by their attributes
A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth · 2009
Earlier work this paper cites.
The graph neural network model
F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini · 2009
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman · 2010
Earlier work this paper cites.
Auto-context and its application to high-level vision tasks and 3d brain image segmentation
Z. Tu and X. Bai · 2010
Earlier work this paper cites.
Discriminative models for multi-class object layout
C. Desai, D. Ramanan, and C. C. Fowlkes · 2011
Earlier work this paper cites.
Efficient inference in fully connected crfs with gaussian edge potentials
P. Krähenbühl and V. Koltun · 2011
Earlier work this paper cites.
Random walk inference and learning in a large scale knowledge base
N. Lao, T. Mitchell, and W. W. Cohen · 2011
Earlier work this paper cites.
Relative attributes
D. Parikh and K. Grauman · 2011
Earlier work this paper cites.
Unbiased look at dataset bias
A. Torralba and A. A. Efros · 2011
Earlier work this paper cites.
Diagnosing error in object detectors
D. Hoiem, Y. Chodpathumwan, and Q. Dai · 2012
Earlier work this paper cites.

Similar

Neil: Extracting visual knowledge from web data
X. Chen, A. Shrivastava, and A. Gupta · 2013
Cited alongside, same era.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio · 2014
Cited alongside, same era.
Large-scale object classification using label relation graphs
J. Deng, N. Ding, Y. Jia, A. Frome, K. Murphy, S. Bengio, Y. Li, H. Neven, and H. Adam · 2014
Cited alongside, same era.
Learning everything about anything: Webly-supervised visual concept learning
S. K. Divvala, A. Farhadi, and C. Guestrin · 2014
Cited alongside, same era.
Microsoft coco: Common objects in context
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Cited alongside, same era.
Visualizing and understanding convolutional networks
M. D. Zeiler and R. Fergus · 2014
Cited alongside, same era.
Deep convolutional networks on graph-structured data
Original
M. Henaff, J. Bruna, and Y. LeCun · 2015
Cited alongside, same era.
Image retrieval using scene graphs
J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei · 2015
Cited alongside, same era.
Gated graph sequence neural networks
Original
Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
Original
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Cited alongside, same era.
U-net: Convolutional networks for biomedical image segmentation
O. Ronneberger, P. Fischer, and T. Brox · 2015
Cited alongside, same era.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Cited alongside, same era.

Then

Semi-supervised classification with graph convolutional networks
Original
T. N. Kipf and M. Welling · 2016
Later among the works it cites.
Visual genome: Connecting language and vision using crowdsourced dense image annotations
Original
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al · 2016
Later among the works it cites.
Feature pyramid networks for object detection
Original
T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie · 2016
Later among the works it cites.
Visual relationship detection with language priors
C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei · 2016
Later among the works it cites.
The more you know: Using knowledge graphs for image classification
Original
K. Marino, R. Salakhutdinov, and A. Gupta · 2016
Later among the works it cites.
Stacked hourglass networks for human pose estimation
A. Newell, K. Yang, and J. Deng · 2016
Later among the works it cites.
Learning convolutional neural networks for graphs
M. Niepert, M. Ahmed, and K. Kutzkov · 2016
Later among the works it cites.
Beyond Skip Connections: Top-Down Modulation for Object Detection
Original
A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta · 2016
Later among the works it cites.
Convolutional pose machines
S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh · 2016
Later among the works it cites.
Ask me anything: Free-form visual question answering based on knowledge from external sources
Q. Wu, P. Wang, C. Shen, A. Dick, and A. van den Hengel · 2016
Later among the works it cites.
Top-down learning for structured labeling with convolutional pseudoprior
S. Xie, X. Huang, and Z. Tu · 2016
Later among the works it cites.
Dynamic memory networks for visual and textual question answering
C. Xiong, S. Merity, and R. Socher · 2016
Later among the works it cites.
Stacked attention networks for image question answering
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola · 2016
Later among the works it cites.
Semantic understanding of scenes through the ade20k dataset
Original
B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba · 2016
Later among the works it cites.
Spatial memory for context reasoning in object detection
Original
X. Chen and A. Gupta · 2017
Later among the works it cites.
A simple neural network module for relational reasoning
Original
A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap · 2017
Later among the works it cites.
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
C. Sun, A. Shrivastava, S. Singh, and A. Gupta · 2017
Later among the works it cites.
Scene graph generation by iterative message passing
Original
D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…