2016

"What happens if..." Learning to Predict the Effect of Forces in Images

Mottaghi, Roozbeh, Rastegari, Mohammad, Gupta, Abhinav et al.

Understand

What happens if one pushes a cup sitting on a table toward the edge of the table? How about pushing a desk against a wall? In this paper, we study the problem of understanding the movements of objects as a result of applying external forces to them.

For a given force vector applied to a specific location in an image, our goal is to predict long-term sequential movements caused by that force.
Doing so entails reasoning about scene geometry, objects, their attributes, and the physical rules that govern the movements of objects.
We design a deep neural network model that learns long-term sequential dependencies of object movements while taking into account the geometry and appearance of the scene by combining Convolutional and Recurrent Neural Networks.

Built on

Bhat, K.S., Seitz, S.M., Popovic, J.: Computing the physical parameters of rigid-body motion from video. In: ECCV (2002)
2002
Earlier work this paper cites.
Murphy, K., Torralba, A., Freeman, W.T.: Using the forest to see the trees: A graphical model relating features, objects, and scenes. In: NIPS (2003)
2003
Earlier work this paper cites.
Heitz, G., Gould, S., Saxena, A., Koller, D.: Cascaded classification models: Combining models for holistic scene understanding. In: NIPS (2008)
2008
Earlier work this paper cites.
Sutskever, I., Hinton, G.E., Taylor, G.W.: The recurrent temporal restricted boltzmann machine. In: NIPS (2008)
2008
Earlier work this paper cites.
Vondrak, M., Sigal, L., Jenkins, O.C.: Physical simulation for probabilistic motion tracking. In: CVPR (2008)
2008
Earlier work this paper cites.
Brubaker, M.A., Sigal, L., Fleet, D.J.: Estimating contact dynamics. In: ICCV (2009)
2009
Earlier work this paper cites.
Li, L.J., Socher, R., Fei-Fei, L.: Towards total scene understanding: Classification, annotation and segmentation in an automatic framework. In: CVPR (2009)
2009
Earlier work this paper cites.
Everingham, M., Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. IJCV (2010)
2010
Earlier work this paper cites.
Gupta, A., Efros, A.A., Hebert, M.: Blocks world revisited: Image understanding using qualitative geometry and mechanics. In: ECCV (2010)
2010
Earlier work this paper cites.
Yuen, J., Torralba, A.: A data-driven approach for event prediction. In: ECCV (2010)
2010
Earlier work this paper cites.
Hamrick, J., Battaglia, P., Tenenbaum., J.B.: Internal physics models guide probabilistic judgments about object dynamics. Annual Meeting of the Cognitive Science Society (2011)
2011
Earlier work this paper cites.
Salzmann, M., Urtasun, R.: Physically-based motion models for 3d tracking: A convex formulation. In: ICCV (2011)
2011
Earlier work this paper cites.
Jiang, Y., Lim, M., Zheng, C., Saxena, A.: Learning to place new objects in a scene. IJRR (2012)
2012
Earlier work this paper cites.
Kitani, K.M., Ziebart, B.D., Bagnell, J.A.D., Hebert, M.: Activity forecasting. In: ECCV (2012)
2012
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
2012
Earlier work this paper cites.

Similar

Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: ECCV (2012)
2012
Cited alongside, same era.
Yao, J., Fidler, S., Urtasun, R.: Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation. In: CVPR (2012)
2012
Cited alongside, same era.
Battaglia, P., Hamrick, J., Tenenbaum., J.B.: Simulation as an engine of physical scene understanding. PNAS (2013)
2013
Cited alongside, same era.
Choi, W., Chao, Y.W., Pantofaru, C., Savarese, S.: Understanding indoor scenes using 3d geometric phrases. In: CVPR (2013)
2013
Cited alongside, same era.
Jia, Z., Gallagher, A., Saxena, A., Chen, T.: 3d-based reasoning with blocks, support, and stability. In: CVPR (2013)
2013
Cited alongside, same era.
Lin, D., Fidler, S., Urtasun, R.: Holistic scene understanding for 3d object detection with rgbd cameras. In: ICCV (2013)
2013
Cited alongside, same era.
Zheng, B., Zhao, Y., Yu, J.C., Ikeuchi, K., Zhu, S.C.: Beyond point clouds: Scene understanding by reasoning geometry and physics. In: CVPR (2013)
2013
Cited alongside, same era.
Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NIPS (2014)
2014
Cited alongside, same era.
Fouhey, D.F., Zitnick, C.: Predicting object dynamics in scenes. In: CVPR (2014)
2014
Cited alongside, same era.
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014)
2014
Cited alongside, same era.
Michalski, V., Memisevic, R., Konda, K.: Modeling deep temporal dependencies with recurrent grammar cells. In: NIPS (2014)
2014
Cited alongside, same era.
Pintea, S.L., van Gemert, J.C., Smeulders, A.W.M.: Déjà vu: - motion prediction in static images. In: ECCV (2014)
2014
Cited alongside, same era.

Then

Zheng, B., Zhao, Y., Yu, J.C., Ikeuchi, K., Zhu, S.C.: Detecting potential falling objects by inferring human action and natural disturbance. In: ICRA (2014)
2014
Later among the works it cites.
Handa, A., Patraucean, V., Badrinarayanan, V., Stent, S., Cipolla, R.: Scenenet: Understanding real world indoor scenes with synthetic data. In: arXiv (2015)
2015
Later among the works it cites.
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015)
2015
Later among the works it cites.
Le, Q.V., Jaitly, N., Hinton, G.E.: A simple way to initialize recurrent networks of rectified linear units. In: ArXiv (2015)
2015
Later among the works it cites.
Levine, S., Finn, C., Darrell, T., Abbeel, P.: End-to-end training of deep visuomotor policies. In: arXiv (2015)
2015
Later among the works it cites.
Oh, J., Guo, X., Lee, H., Lewis, R.L., Singh, S.P.: Action-conditional video prediction using deep networks in atari games. In: NIPS (2015)
2015
Later among the works it cites.
Rastegari, M., Keskin, C., Kohli, P., Izadi, S.: Computationally bounded retrieval. In: CVPR (2015)
2015
Later among the works it cites.
Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: A rgb-d scene understanding benchmark suite. In: CVPR (2015)
2015
Later among the works it cites.
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: CVPR (2015)
2015
Later among the works it cites.
Walker, J., Gupta, A., Hebert, M.: Dense optical flow prediction from a static image. In: ICCV (2015)
2015
Later among the works it cites.
Wang, X., Fouhey, D.F., Gupta, A.: Designing deep networks for surface normal estimation. In: CVPR (2015)
2015
Later among the works it cites.
Wu, J., Yildirim, I., Lim, J.J., Freeman, W.T., Tenenbaum, J.B.: Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In: NIPS (2015)
2015
Later among the works it cites.
Fragkiadaki, K., Agrawal, P., Levine, S., Malik, J.: Learning predictive visual models of physics for playing billiards. In: ICLR (2016)
2016
Closest in time.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
2016
Closest in time.
Mottaghi, R., Bagherinezhad, H., Rastegari, M., Farhadi, A.: Newtonian image understanding: Unfolding the dynamics of objects in static images. In: CVPR (2016)
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

&#34;What happens if...&#34; Learning to Predict the Effect of Forces in Images

Understand

Built on

Similar

Then

Beyond the bibliography

Built on

Similar

Then

Beyond the bibliography

"What happens if..." Learning to Predict the Effect of Forces in Images