2019

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

Yi, Kexin, Gan, Chuang, Li, Yunzhu et al.

Understand

The ability to reason about temporal and causal events from videos lies at the core of human intelligence.

Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure.
We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance.
To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks.

Built on

Long Short-Term Memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Core Knowledge
Elizabeth S. Spelke · 2000
Earlier work this paper cites.
Causality
Judea Pearl · 2009
Earlier work this paper cites.
Bullet Physics Engine
Erwin Coumans · 2010
Earlier work this paper cites.
Simulation As an Engine of Physical Scene Understanding
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum · 2013
Earlier work this paper cites.
Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition
Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko · 2013
Earlier work this paper cites.
Distributed representations of words and phrases and their compositionality
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean · 2013
Earlier work this paper cites.
Imagining the Unseen: Stability-Based Cuboid Arrangements for Scene Understanding
Tianjia Shao, Aron Monszpart, Youyi Zheng, Bongjin Koo, Weiwei Xu, Kun Zhou, and Niloy J. Mitra · 2014
Earlier work this paper cites.
VQA: Visual Question Answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh · 2015
Earlier work this paper cites.
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2015
Earlier work this paper cites.
Activitynet: A large-scale video benchmark for human activity understanding
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles · 2015
Earlier work this paper cites.
How, whether, why: Causal judgments as counterfactual contrasts
Tobias Gerstenberg, Noah D Goodman, David A Lagnado, and Joshua B Tenenbaum · 2015
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Convolutional lstm network: A machine learning approach for precipitation nowcasting
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo · 2015
Earlier work this paper cites.
On the Nature and Origin of Intuitive Theories: Learning, Physics and Psychology
Tomer David Ullman · 2015
Earlier work this paper cites.
Sequence to sequence-video to text
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko · 2015
Earlier work this paper cites.
Learning to Poke by Poking: Experiential Learning of Intuitive Physics
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine · 2016
Earlier work this paper cites.
Interaction Networks for Learning about Objects, Relations and Physics
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu · 2016
Earlier work this paper cites.
Blender - a 3D modelling and rendering package
Blender Online Community · 2016
Earlier work this paper cites.
Unsupervised Learning for Physical Interaction through Video Prediction
Chelsea Finn, Ian Goodfellow, and Sergey Levine · 2016
Earlier work this paper cites.
Learning perceptual causality from video
Amy Fire and Song-Chun Zhu · 2016
Earlier work this paper cites.
Learning Visual Predictive Models of Physics for Playing Billiards
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik · 2016
Earlier work this paper cites.

Similar

Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Learning Physical Intuition of Block Towers by Example
Adam Lerer, Sam Gross, and Rob Fergus · 2016
Cited alongside, same era.
“What happens if…” learning to predict the effect of forces in images
Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi · 2016
Cited alongside, same era.
MovieQA: Understanding Stories in Movies through Question-Answering
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler · 2016
Cited alongside, same era.
Stacked Attention Networks for Image Question Answering
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola · 2016
Cited alongside, same era.
Visual7w: Grounded question answering in images
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei · 2016
Cited alongside, same era.
A Compositional Object-Based Approach to Learning Physical Dynamics
Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum · 2017
Cited alongside, same era.
Self-supervised visual planning with temporal skip connections
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine · 2017
Cited alongside, same era.
Semantic compositional networks for visual captioning
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng · 2017
Cited alongside, same era.
Tall: Temporal activity localization via language query
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia · 2017
Cited alongside, same era.
Mask R-CNN
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Cited alongside, same era.
Localizing moments in video with natural language
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell · 2017
Cited alongside, same era.

Then

Explicit Reasoning Over End-to-End Neural Architectures for Visual Question Answering
Somak Aditya, Yezhou Yang, and Chitta Baral · 2018
Later among the works it cites.
Learning Interpretable Spatial Operations in a Rich 3D Blocks World
Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu · 2018
Later among the works it cites.
Visual Question Reasoning on General Dependency Tree
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin · 2018
Later among the works it cites.
Compositional Attention Networks for Machine Reasoning
Drew A. Hudson and Christopher D. Manning · 2018
Later among the works it cites.
TVQA: Localized, compositional video question answering
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg · 2018
Later among the works it cites.
Transparency by Design: Closing the Gap between Performance and Interpretability in Visual Reasoning
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar · 2018
Later among the works it cites.
Learning by asking questions
Ishan Misra, Ross Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten · 2018
Later among the works it cites.
FiLM: Visual Reasoning with a General Conditioning Layer
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville · 2018
Later among the works it cites.
DDRprog: A Clevr Differentiable Dynamic Reasoning Programmer
Original
Joseph Suarez, Justin Johnson, and Fei-Fei Li · 2018
Later among the works it cites.
A Dataset and Architecture for Visual Reasoning with a Working Memory
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo · 2018
Later among the works it cites.
Interpretable intuitive physics model
Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta · 2018
Later among the works it cites.
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum · 2018
Later among the works it cites.
Monet: Unsupervised scene decomposition and representation
Original
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner · 2019
Closest in time.
Heterogeneous memory enhanced multimodal attention model for video question answering
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang · 2019
Closest in time.
GQA: A new dataset for real-world visual reasoning and compositional question answering
Drew A Hudson and Christopher D Manning · 2019
Closest in time.
Propagation Networks for Model-Based Control under Partial Observation
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, and Russ Tedrake · 2019
Closest in time.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu · 2019
Closest in time.
Social-IQ: A question answering benchmark for artificial social intelligence
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency · 2019
Closest in time.
From recognition to cognition: Visual commonsense reasoning
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi · 2019
Closest in time.
Cophy: Counterfactual learning of physical dynamics
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf · 2020
Closest in time.
Cater: A diagnostic dataset for compositional actions and temporal reasoning
Rohit Girdhar and Deva Ramanan · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…