2021

Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution

Zhang, Chi, Jia, Baoxiong, Zhu, Song-Chun et al.

Understand

Spatial-temporal reasoning is a challenging task in Artificial Intelligence (AI) due to its demanding but unique nature: a theoretic requirement on representing and reasoning based on spatial-temporal knowledge in mind, and an applied requirement on a high-level cognitive system capable of navigating and acting in space and time.

Recent works have focused on an abstract reasoning task of this kind -- Raven's Progressive Matrices (RPM).
Despite the encouraging progress on RPM that achieves human-level performance in terms of accuracy, modern approaches have neither a treatment of human-like reasoning on generalization, nor a potential to generate answers.
To fill in this gap, we propose a neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner; central to the PrAE learner is the process of probabilistic abduction and execution on a probabilistic scene representation, akin to the mental manipulation of objects.

Built on

The abilities of man
Charles Spearman · 1927
Earlier work this paper cites.
Mental tests used in genetic studies: The performance of related individuals on tests mainly educative and mainly reproductive
James C Raven · 1936
Earlier work this paper cites.
Quote the Raven? Nevermore
Earl Hunt · 1974
Earlier work this paper cites.
Lectures in pattern theory i, ii and iii: Pattern analysis, pattern synthesis and regular structures, 1976
Ulf Grenander · 1976
Earlier work this paper cites.
The topography of ability and learning correlations
R E Snow, Patrick Kyllonen, and B Marshalek · 1984
Earlier work this paper cites.
Connectionism and cognitive architecture: A critical analysis
Jerry A Fodor, Zenon W Pylyshyn, et al · 1988
Earlier work this paper cites.
What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test
Patricia A Carpenter, Marcel A Just, and Peter Shell · 1990
Earlier work this paper cites.
Divergence measures based on the shannon entropy
Jianhua Lin · 1991
Earlier work this paper cites.
Forward models: Supervised learning with a distal teacher
Michael I Jordan and David E Rumelhart · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Ronald J Williams · 1992
Earlier work this paper cites.
Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought
Douglas R Hofstadter · 1995
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al · 1998
Earlier work this paper cites.
Rethinking eliminative connectionism
Gary F Marcus · 1998
Earlier work this paper cites.
Raven’s progressive matrices and vocabulary scales
John C Raven and John Hugh Court · 1998
Earlier work this paper cites.
Introduction to reinforcement learning
Richard S Sutton, Andrew G Barto, et al · 1998
Earlier work this paper cites.
Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling
Song-Chun Zhu, Yingnian Wu, and David Mumford · 1998
Earlier work this paper cites.
Automated Planning: theory and practice
Malik Ghallab, Dana Nau, and Paolo Traverso · 2004
Earlier work this paper cites.
Core knowledge of geometry in an amazonian indigene group
Stanislas Dehaene, Véronique Izard, Pierre Pica, and Elizabeth Spelke · 2006
Earlier work this paper cites.
Thinking in pictures: And other reports from my life with autism
Temple Grandin · 2006
Earlier work this paper cites.
Vision as bayesian inference: analysis by synthesis?
Alan Yuille and Daniel Kersten · 2006
Earlier work this paper cites.
A stochastic grammar of images
Song-Chun Zhu and David Mumford · 2007
Earlier work this paper cites.
Improving fluid intelligence with training on working memory
Susanne M Jaeggi, Martin Buschkuehl, John Jonides, and Walter J Perrig · 2008
Earlier work this paper cites.
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston · 2009
Earlier work this paper cites.
Solving geometric analogy problems through two-stage analogical mapping
Andrew Lovett, Emmett Tomai, Kenneth Forbus, and Jeffrey Usher · 2009
Earlier work this paper cites.
A structure-mapping model of raven’s progressive matrices
Andrew Lovett, Kenneth Forbus, and Jeffrey Usher · 2010
Earlier work this paper cites.
Flexible intuitions of euclidean geometry in an amazonian indigene group
Véronique Izard, Pierre Pica, Elizabeth S Spelke, and Stanislas Dehaene · 2011
Earlier work this paper cites.

Similar

The Oxford handbook of thinking and reasoning
Keith James Holyoak and Robert G Morrison · 2012
Cited alongside, same era.
A bayesian model of rule induction in raven’s progressive matrices
Daniel R Little, Stephan Lewandowsky, and Thomas L Griffiths · 2012
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Opendr: An approximate differentiable renderer
Matthew M Loper and Michael J Black · 2014
Cited alongside, same era.
Confident reasoning on raven’s progressive matrices tests
Keith McGreggor and Ashok Goel · 2014
Cited alongside, same era.
Fractals and ravens
Keith McGreggor, Maithilee Kunda, and Ashok Goel · 2014
Cited alongside, same era.
Vqa: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh · 2015
Cited alongside, same era.
Symbol acquisition for probabilistic high-level planning
George Konidaris, Leslie Kaelbling, and Tomas Lozano-Perez · 2015
Cited alongside, same era.
Automatic generation of raven’s progressive matrices
Ke Wang and Zhendong Su · 2015
Cited alongside, same era.
A theory of generative convnet
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu · 2016
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
Tgif-qa: Toward spatio-temporal reasoning in visual question answering
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim · 2017
Cited alongside, same era.

Then

Neural-symbolic vqa: Disentangling reasoning from vision and language understanding
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum · 2018
Later among the works it cites.
Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense
Yixin Chen, Siyuan Huang, Tao Yuan, Yixin Zhu, Siyuan Qi, and Song-Chun Zhu · 2019
Later among the works it cites.
Decomposing human causal learning: Bottom-up associative learning and top-down schema reasoning
Mark Edmonds, Siyuan Qi, Yixin Zhu, James Kubricht, Song-Chun Zhu, and Hongjing Lu · 2019
Later among the works it cites.
Visual concept-metaconcept learning
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu · 2019
Later among the works it cites.
Divergence triangle for joint training of generator model, energy-based model, and inferential model
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu · 2019
Later among the works it cites.
Learning to make analogies by contrasting abstract relational structure
Felix Hill, Adam Santoro, David GT Barrett, Ari S Morcos, and Timothy Lillicrap · 2019
Later among the works it cites.
Continuous relaxation of symbolic planner for one-shot imitation learning
De-An Huang, Danfei Xu, Yuke Zhu, Animesh Garg, Silvio Savarese, Li Fei-Fei, and Juan Carlos Niebles · 2019
Later among the works it cites.
The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu · 2019
Later among the works it cites.
Rebooting AI: building artificial intelligence we can trust
Gary Marcus and Ernest Davis · 2019
Later among the works it cites.
Learning energy-based spatial-temporal generative convnets for dynamic patterns
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu · 2019
Later among the works it cites.
Raven: A dataset for relational and analogical visual reasoning
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu · 2019
Later among the works it cites.
Learning perceptual inference by contrasting
Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu · 2019
Later among the works it cites.
Metastyle: Three-way trade-off among speed, flexibility, and quality in neural style transfer
Chi Zhang, Yixin Zhu, and Song-Chun Zhu · 2019
Later among the works it cites.
Abstract reasoning with distracting features
Kecheng Zheng, Zheng-Jun Zha, and Wei Wei · 2019
Later among the works it cites.
Theory-based causal transfer: Integrating instance-level induction and abstract-level structure learning
Mark Edmonds, Xiaojian Ma, Siyuan Qi, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu · 2020
Later among the works it cites.
Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning
Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu · 2020
Later among the works it cites.
Insights for ai from the human mind
Gary Marcus and Ernest Davis · 2020
Later among the works it cites.
A generalized earley parser for human activity parsing and prediction
Siyuan Qi, Baoxiong Jia, Siyuan Huang, Ping Wei, and Song-Chun Zhu · 2020
Later among the works it cites.
A closer look at generalisation in raven
Steven Spratley, Krista Ehinger, and Tim Miller · 2020
Later among the works it cites.
Abstract diagrammatic reasoning with multiplex graph networks
Duo Wang, Mateja Jamnik, and Pietro Lio · 2020
Later among the works it cites.
Clevrer: Collision events for video representation and reasoning
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum · 2020
Later among the works it cites.
Machine number sense: A dataset of visual arithmetic problems for abstract and relational reasoning
Wenhe Zhang, Chi Zhang, Yixin Zhu, and Song-Chun Zhu · 2020
Later among the works it cites.
Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense
Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al · 2020
Later among the works it cites.
Stratified rule-aware network for abstract visual reasoning
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai · 2021
Closest in time.
Halma: Humanlike abstraction learning meets affordance in rapid problem solving
Original
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu · 2021
Closest in time.
Acre: Abstract causal reasoning beyond covariation
Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…