2018

Videos as Space-Time Region Graphs

Wang, Xiaolong, Gupta, Abhinav

Understand

How do humans recognize the action "opening a book" ? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects.

In this paper, we propose to represent videos as space-time region graphs which capture these two important cues.
Our graph nodes are defined by the object region proposals from different frames in a long range video.
These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects.

Built on

A survey of the statistical theory of shape
Kendall, D.G.: · 1989
Earlier work this paper cites.
Conditional random fields: Probabilistic models for segmenting and labeling sequence data
Lafferty, J., McCallum, A., Pereira, F.C.: · 2001
Earlier work this paper cites.
On space-time interest points
Laptev, I.: · 2005
Earlier work this paper cites.
Human detection using oriented histograms of flow and appearance
Dalal, N., Triggs, B., Schmid, C.: · 2006
Earlier work this paper cites.
Using multiple segmentations to discover objects and their extent in image collections
Russell, B.C., Freeman, W.T., Efros, A.A., Sivic, J., Zisserman, A.: · 2006
Earlier work this paper cites.
A spatio-temporal descriptor based on 3d-gradients
Klaser, A., Marszalek, M., Schmid, C.: · 2008
Earlier work this paper cites.
Learning realistic human actions from movies
Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: · 2008
Earlier work this paper cites.
Observing human-object interactions: Using spatial and functional compatibility for recognition
Gupta, A., Kembhavi, A., Davis, L.S.: · 2009
Earlier work this paper cites.
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: · 2009
Earlier work this paper cites.
Modeling mutual context of object and human pose in human-object interaction activities
Yao, B., Fei-Fei, L.: · 2010
Earlier work this paper cites.
Convolutional learning of spatio-temporal features
Taylor, G.W., Fergus, R., LeCun, Y., Bregler, C.: · 2010
Earlier work this paper cites.
Efficiently selecting regions for scene understanding
Kumar, M.P., Koller, D.: · 2010
Earlier work this paper cites.
Hidden part models for human action recognition: Probabilistic vs. max-margin
Wang, Y., Mori, G.: · 2011
Earlier work this paper cites.
Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis
Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: · 2011
Earlier work this paper cites.
Efficient inference in fully connected crfs with gaussian edge potentials
Krähenbühl, P., Koltun, V.: · 2011
Earlier work this paper cites.
Efficient inference in fully connected crfs with gaussian edge potentials
Krähenbühl, P., Koltun, V.: · 2011
Earlier work this paper cites.
Learning spatiotemporal graphs of human activities
Brendel, W., Todorovic, S.: · 2011
Earlier work this paper cites.
Action bank: A high-level representation of activity in video
Sadanand, S., Corso, J.J.: · 2012
Earlier work this paper cites.
Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation
Yao, J., Fidler, S., Urtasun, R.: · 2012
Earlier work this paper cites.
Efficient activity detection with max-subgraph search
Chen, C.Y., Grauman, K.: · 2012
Earlier work this paper cites.
Improving neural networks by preventing co-adaptation of feature detectors
Original
Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: · 2012
Earlier work this paper cites.
Action recognition with improved trajectories
Wang, H., Schmid, C.: · 2013
Earlier work this paper cites.
Action recognition with actons
Zhu, J., Wang, B., Yang, X., Zhang, W., Tu, Z.: · 2013
Earlier work this paper cites.
3d convolutional neural networks for human action recognition
Ji, S., Xu, W., Yang, M., Yu, K.: · 2013
Earlier work this paper cites.
Two-stream convolutional networks for action recognition in videos
Simonyan, K., Zisserman, A.: · 2014
Earlier work this paper cites.
Action recognition with stacked fisher vectors
Peng, X., Zou, C., Qiao, Y., Peng, Q.: · 2014
Earlier work this paper cites.
Large-scale video classification with convolutional neural networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: · 2014
Earlier work this paper cites.
Semantic image segmentation with deep convolutional nets and fully connected CRFs
Original
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: · 2014
Earlier work this paper cites.
Beyond short snippets: Deep networks for video classification
Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: · 2015
Earlier work this paper cites.
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L.A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., Darrell, T.: · 2015
Earlier work this paper cites.

Similar

Learning spatiotemporal features with 3d convolutional networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: · 2015
Cited alongside, same era.
Contextual action recognition with r*cnn
Gkioxari, G., Girshick, R., Malik, J.: · 2015
Cited alongside, same era.
Beyond gaussian pyramid: Multi-skip feature stacking for action recognition
Lan, Z., Lin, M., Li, X., Hauptmann, A.G., Raj, B.: · 2015
Cited alongside, same era.
Action recognition with trajectory-pooled deep-convolutional descriptors
Wang, L., Qiao, Y., Tang, X.: · 2015
Cited alongside, same era.
Unsupervised learning of video representations using lstms
Original
Srivastava, N., Mansimov, E., Salakhutdinov, R.: · 2015
Cited alongside, same era.
Temporal localization of fine-grained actions in videos by domain transfer from web images
Sun, C., Shetty, S., Sukthankar, R., Nevatia, R.: · 2015
Cited alongside, same era.
Modeling spatial-temporal clues in a hybrid deep learning framework for video classification
Original
Wu, Z., Wang, X., Jiang, Y.G., Ye, H., Xue, X.: · 2015
Cited alongside, same era.
Learning spatiotemporal features with 3d convolutional networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: · 2015
Cited alongside, same era.
Actions and attributes from wholes and parts
Gkioxari, G., Girshick, R., Malik, J.: · 2015
Cited alongside, same era.
Conditional random fields as recurrent neural networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H.: · 2015
Cited alongside, same era.
Fully connected deep structured networks
Original
Schwing, A.G., Urtasun, R.: · 2015
Cited alongside, same era.
Faster R-CNN: Towards real-time object detection with region proposal networks
Ren, S., He, K., Girshick, R., Sun, J.: · 2015
Cited alongside, same era.

Then

Quo vadis, action recognition? a new model and the kinetics dataset
Carreira, J., Zisserman, A.: · 2017
Later among the works it cites.
Rethinking spatiotemporal feature learning for video understanding
Original
Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: · 2017
Later among the works it cites.
What actions are needed for understanding human actions in videos?
Sigurdsson, G.A., Russakovsky, O., Gupta, A.: · 2017
Later among the works it cites.
Semi-supervised classification with graph convolutional networks
Kipf, T.N., Welling, M.: · 2017
Later among the works it cites.
The ”something something” video database for learning and evaluating visual common sense
Original
Goyal, R., Kahou, S.E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fründ, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., Memisevic, R.: · 2017
Later among the works it cites.
Temporal relational reasoning in videos
Zhou, B., Andonian, A., Torralba, A.: · 2017
Later among the works it cites.
Revisiting the effectiveness of off-the-shelf temporal modeling approaches for large-scale video classification
Original
Bian, Y., Gan, C., Liu, X., Li, F., Long, X., Li, Y., Qi, H., Zhou, J., Wen, S., Lin, Y.: · 2017
Later among the works it cites.
Aggregated residual transformations for deep neural networks
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: · 2017
Later among the works it cites.
Learning spatio-temporal representation with pseudo-3d residual networks
Qiu, Z., Yao, T., Mei, T.: · 2017
Later among the works it cites.
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., Lillicrap, T.: · 2017
Later among the works it cites.
Visual interaction networks
Watters, N., Tacchetti, A., Weber, T., Pascanu, R., Battaglia, P., Zoran, D.: · 2017
Later among the works it cites.
Dense and low-rank Gaussian CRFs using deep embeddings
Chandra, S., Usunier, N., Kokkinos, I.: · 2017
Later among the works it cites.
Segmentation-aware convolutional networks using local attention masks
Harley, A., Derpanis, K., Kokkinos, I.: · 2017
Later among the works it cites.
Learning affinity via spatial propagation networks
Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M.H., Kautz, J.: · 2017
Later among the works it cites.
The more you know: Using knowledge graphs for image classification
Marino, K., Salakhutdinov, R., Gupta, A.: · 2017
Later among the works it cites.
Joint discovery of object states and manipulation actions
Alayrac, J.B., Sivic, J., Laptev, I., Lacoste-Julien, S.: · 2017
Later among the works it cites.
Scc: Semantic context cascade for efficient action detection
Heilbron, F.C., Barrios, W., Escorcia, V., Ghanem, B.: · 2017
Later among the works it cites.
Temporal dynamic graph lstm for action-driven video object detection
Yuan, Y., Liang, X., Wang, X., Yeung, D.Y., Gupta, A.: · 2017
Later among the works it cites.
Mask R-CNN
He, K., Gkioxari, G., Dollár, P., Girshick, R.: · 2017
Later among the works it cites.
Feature pyramid networks for object detection
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: · 2017
Later among the works it cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: · 2017
Later among the works it cites.
The kinetics human action video dataset
Original
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: · 2017
Later among the works it cites.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Original
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: · 2017
Later among the works it cites.
Asynchronous temporal fields for action recognition
Sigurdsson, G.A., Divvala, S., Farhadi, A., Gupta, A.: · 2017
Later among the works it cites.
A closer look at spatiotemporal convolutions for action recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: · 2018
Closest in time.
Relation networks for object detection
Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: · 2018
Closest in time.
Detecting and recognizing human-object intaractions
Gkioxari, G., Girshick, R., Dollár, P., He, K.: · 2018
Closest in time.
Attend and interact: Higher-order object interactions for video understanding
Ma, C.Y., Kadav, A., Melvin, I., Kira, Z., AlRegib, G., Graf, H.P.: · 2018
Closest in time.
Non-local neural networks
Wang, X., Girshick, R., Gupta, A., He, K.: · 2018
Closest in time.
Spatial temporal graph convolutional networks for skeleton-based action recognition
Yan, S., Xiong, Y., Lin, D.: · 2018
Closest in time.
Detectron
Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P., He, K.: · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…