2015

Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries

Zhu, Yuke, Zhang, Ce, Ré, Christopher et al.

Understand

The complexity of the visual world creates significant challenges for comprehensive visual understanding.

In spite of recent successes in visual recognition, today's vision systems would still struggle to deal with visual queries that require a deeper reasoning.
We propose a knowledge base (KB) framework to handle an assortment of visual queries, without the need to train new classifiers for new tasks.
Building such a large-scale multimodal KB presents a major challenge of scalability.

Built on

Cyc: A large-scale investment in knowledge infrastructure
D. B. Lenat · 1995
Earlier work this paper cites.
Learning probabilistic relational models
L. Getoor, N. Friedman, D. Koller, and A. Pfeffer · 1999
Earlier work this paper cites.
Factor graphs and the sum-product algorithm
F. Kschischang, B. Frey, and H.-A. Loeliger · 2001
Earlier work this paper cites.
Training products of experts by minimizing contrastive divergence
G. E. Hinton · 2002
Earlier work this paper cites.
Continuous restricted boltzmann machine with an implementable training algorithm
H. Chen and A. F. Murray · 2003
Earlier work this paper cites.
Discriminative random fields: A discriminative framework for contextual interaction in classification
S. Kumar and M. Hebert · 2003
Earlier work this paper cites.
Multiscale conditional random fields for image labeling
X. He, R. S. Zemel, and M. Á. Carreira-Perpiñán · 2004
Earlier work this paper cites.
On contrastive divergence learning
M. Á. Carreira-Perpiñán and G. E. Hinton · 2005
Earlier work this paper cites.
Developing the american time use survey activity classification system
K. Shelley · 2005
Earlier work this paper cites.
Contextual models for object detection using boosted random fields
A. Torralba, K. Murphy, and W. Freeman · 2005
Earlier work this paper cites.
A simple bayesian framework for content-based image retrieval
K. Heller and Z. Ghahramani · 2006
Earlier work this paper cites.
Putting objects in perspective
D. Hoiem, A. A. Efros, and M. Hebert · 2006
Earlier work this paper cites.
Markov logic networks
M. Richardson and P. Domingos · 2006
Earlier work this paper cites.
Freebase: A collaboratively created graph database for structuring human knowledge
K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor · 2008
Earlier work this paper cites.
Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient
T. Tieleman · 2008
Earlier work this paper cites.
Discriminative models for multi-class object layout
C. Desai, D. Ramanan, and C. Fowlkes · 2009
Earlier work this paper cites.
Probabilistic graphical models: principles and techniques
D. Koller and N. Friedman · 2009
Earlier work this paper cites.
StatSnowball: a statistical approach to extracting entity relationships
J. Zhu, Z. Nie, X. Liu, B. Zhang, and J.-R. Wen · 2009
Earlier work this paper cites.
Building Watson: An overview of the DeepQA project
D. Ferrucci et al · 2010
Earlier work this paper cites.
Graph cut based inference with co-occurrence statistics
L. Ladicky, C. Russell, P. Kohli, and P. Torr · 2010
Earlier work this paper cites.

Similar

Scalable probabilistic databases with factor graphs and mcmc
M. Wick, A. McCallum, and G. Miklau · 2010
Cited alongside, same era.
SUN database: Large-scale scene recognition from abbey to zoo
J. Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba · 2010
Cited alongside, same era.
Modeling mutual context of object and human pose in human-object interaction activities
B. Yao and L. Fei-Fei · 2010
Cited alongside, same era.
Tuffy: Scaling up statistical inference in Markov logic networks using an RDBMS
F. Niu, C. Ré, A. Doan, and J. Shavlik · 2011
Cited alongside, same era.
Hogwild!: A lock-free approach to parallelizing stochastic gradient descent
F. Niu, B. Recht, C. Ré, and S. J. Wright · 2011
Cited alongside, same era.
Image ranking and retrieval based on multi-attribute queries
B. Siddiquie, R. S. Feris, and L. S. Davis · 2011
Cited alongside, same era.
Probabilistic Databases
D. Suciu, D. Olteanu, C. Ré, and C. Koch · 2011
Cited alongside, same era.
Exploring tiny images: The roles of appearance and contextual information for machine and human object recognition
D. Parikh, C. L. Zitnick, and T. Chen · 2012
Cited alongside, same era.
Semantic parsing on Freebase from question-answer pairs
J. Berant et al · 2013
Cited alongside, same era.
NEIL: Extracting visual knowledge from web data
X. Chen, A. Shrivastava, and A. Gupta · 2013
Cited alongside, same era.
Grounded compositional semantics for finding and describing images with sentences
R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng · 2013
Cited alongside, same era.
Scene parsing by integrating function, geometry and appearance models
Y. Zhao and S.-C. Zhu · 2013
Cited alongside, same era.

Then

The SUN attribute database: Beyond categories for deeper scene understanding
G. Patterson, C. Xu, H. Su, and J. Hays · 2014
Later among the works it cites.
Sequence to sequence learning with neural networks
I. Sutskever, O. Vinyals, and Q. Le · 2014
Later among the works it cites.
Joint video and text parsing for understanding events and answering queries
K. Tu, M. Meng, M. W. Lee, T. E. Choe, and S.-C. Zhu · 2014
Later among the works it cites.
Visualizing and understanding convolutional networks
M. Zeiler and R. Fergus · 2014
Later among the works it cites.
DimmWitted: A study of main-memory statistical analytics
C. Zhang and C. Ré · 2014
Later among the works it cites.
Learning Deep Features for Scene Recognition using Places Database
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva · 2014
Later among the works it cites.
Reasoning about object affordances in a knowledge base representation
Y. Zhu, A. Fathi, and L. Fei-Fei · 2014
Later among the works it cites.
VQA: Visual question answering
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, L. Zitnick, and D. Parikh · 2015
Closest in time.
Mining semantic affordances of visual object categories
Y.-W. Chao, Z. Wang, R. Mihalcea, and J. Deng · 2015
Closest in time.
Mind’s eye: A recurrent visual representation for image caption generation
X. Chen and C. L. Zitnick · 2015
Closest in time.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Closest in time.
Are you talking to a machine? dataset and methods for multilingual image question answering
H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu · 2015
Closest in time.
Deep visual-semantic alignments for generating image descriptions
A. Karpathy and L. Fei-Fei · 2015
Closest in time.
Generating multi-sentence lingual descriptions of indoor scenes
D. Lin, C. Kong, S. Fidler, and R. Urtasun · 2015
Closest in time.
Don’t just listen, use your imagination: Leveraging visual common sense for non-visual tasks
X. Lin and D. Parikh · 2015
Closest in time.
Ask your neurons: A neural-based approach to answering questions about images
M. Malinowski, M. Rohrbach, and M. Fritz · 2015
Closest in time.
Viske: Visual knowledge extraction and question answering by visual verification of relation phrases
F. Sadeghi, S. K. Divvala, and A. Farhadi · 2015
Closest in time.
Show and tell: A neural image caption generator
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan · 2015
Closest in time.
Show, attend and tell: Neural image caption generation with visual attention
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio · 2015
Closest in time.
Visual Madlibs: Fill in the blank Image Generation and Question Answering
L. Yu, E. Park, A. C. Berg, and T. L. Berg · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…