2020

Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks

Plizzari, Chiara, Cannici, Marco, Matteucci, Matteo

Understand

Skeleton-based Human Activity Recognition has achieved great interest in recent years as skeleton data has demonstrated being robust to illumination changes, body scales, dynamic camera views, and complex background.

In particular, Spatial-Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs.
Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially when it comes to extracting effective information from joint motion patterns and their correlations.
In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator.

Built on

Dropattention: A regularization method for fully-connected self-attention networks
Original
Zehui, L., Liu, P., Huang, L., Fu, J., Chen, J., Qiu, X., Huang, X., 2019 · 1907
Earlier work this paper cites.
Transformers without tears: Improving the normalization of self-attention
Original
Nguyen, T.Q., Salazar, J., 2019 · 1910
Earlier work this paper cites.
A survey on 3d skeleton-based action recognition using learning method
Original
Ren, B., Liu, M., Ding, R., Liu, H., 2020 · 2002
Earlier work this paper cites.
A new model for learning in graph domains
Gori, M., Monfardini, G., Scarselli, F., 2005 · 2005
Earlier work this paper cites.
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G., 2008 · 2008
Earlier work this paper cites.
Neural network for graphs: A contextual constructive approach
Micheli, A., 2009 · 2009
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2020 · 2010
Earlier work this paper cites.
Human activity analysis: A review
Aggarwal, J.K., Ryoo, M.S., 2011 · 2011
Earlier work this paper cites.
Parameter efficient multimodal transformers for video representation learning
Original
Lee, S., Yu, Y., Kim, G., Breuel, T., Kautz, J., Song, Y., 2020 · 2012
Earlier work this paper cites.
Microsoft kinect sensor and its effect
Zhang, Z., 2012 · 2012
Earlier work this paper cites.
Point transformer
Original
Zhao, H., Jiang, L., Jia, J., Torr, P., Koltun, V., 2020 · 2012
Earlier work this paper cites.
Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations
Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M., 2013 · 2013
Earlier work this paper cites.
Spectral networks and locally connected networks on graphs
Bruna, J., Zaremba, W., Szlam, A., Lecun, Y., 2014 · 2014
Earlier work this paper cites.
Two-stream convolutional networks for action recognition in videos
Simonyan, K., Zisserman, A., 2014 · 2014
Earlier work this paper cites.
Human action recognition by representing 3d skeletons as points in a lie group
Vemulapalli, R., Arrate, F., Chellappa, R., 2014 · 2014
Earlier work this paper cites.
P-cnn: Pose-based cnn features for action recognition
Chéron, G., Laptev, I., Schmid, C., 2015 · 2015
Earlier work this paper cites.
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., Wang, L., 2015 · 2015
Earlier work this paper cites.
Deep convolutional networks on graph-structured data
Original
Henaff, M., Bruna, J., LeCun, Y., 2015 · 2015
Earlier work this paper cites.
Jointly learning heterogeneous features for rgb-d activity recognition
Hu, J.F., Zheng, W.S., Lai, J., Zhang, J., 2015 · 2015
Earlier work this paper cites.

Similar

Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Ioffe, S., Szegedy, C., 2015 · 2015
Cited alongside, same era.
Convolutional neural networks on graphs with fast localized spectral filtering
Defferrard, M., Bresson, X., Vandergheynst, P., 2016 · 2016
Cited alongside, same era.
Rnn fisher vectors for action recognition and image annotation
Lev, G., Sadeh, G., Klein, B., Wolf, L., 2016 · 2016
Cited alongside, same era.
Spatio-temporal lstm with trust gates for 3d human action recognition
Liu, J., Shahroudy, A., Xu, D., Wang, G., 2016 · 2016
Cited alongside, same era.
Conditional image generation with pixelcnn decoders
Original
Oord, A.v.d., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., Kavukcuoglu, K., 2016 · 2016
Cited alongside, same era.
Ntu rgb+d: A large scale dataset for 3d human activity analysis
Shahroudy, A., Liu, J., Ng, T.T., Wang, G., 2016 · 2016
Cited alongside, same era.
Geometric deep learning: going beyond euclidean data
Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P., 2017 · 2017
Cited alongside, same era.
Investigation of different skeleton features for cnn-based 3d action recognition
Ding, Z., Wang, P., Ogunbona, P.O., Li, W., 2017 · 2017
Cited alongside, same era.
The kinetics human action video dataset
Original
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al., 2017 · 2017
Cited alongside, same era.
Intel realsense stereoscopic depth cameras, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–10
Keselman, L., Iselin Woodfill, J., Grunnet-Jepsen, A., Bhowmik, A., 2017 · 2017
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
Kipf, T.N., Welling, M., 2017 · 2017
Cited alongside, same era.
Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn
Li, B., Dai, Y., Cheng, X., Chen, H., Lin, Y., He, M., 2017 · 2017
Cited alongside, same era.

Then

Recognizing human actions as the evolution of pose estimation maps
Liu, M., Yuan, J., 2018 · 2018
Later among the works it cites.
Image transformer, in: International Conference on Machine Learning, PMLR. pp. 4055–4064
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., Tran, D., 2018 · 2018
Later among the works it cites.
Non-local neural networks, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794–7803
Wang, X., Girshick, R., Gupta, A., He, K., 2018 · 2018
Later among the works it cites.
Spatial temporal graph convolutional networks for skeleton-based action recognition
Yan, S., Xiong, Y., Lin, D., 2018 · 2018
Later among the works it cites.
Openpose: Realtime multi-person 2d pose estimation using part affinity fields
Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S., Sheikh, Y.A., 2019 · 2019
Later among the works it cites.
Ccnet: Criss-cross attention for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603–612
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., Liu, W., 2019 · 2019
Later among the works it cites.
Actional-structural graph convolutional networks for skeleton-based action recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595–3603
Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., Tian, Q., 2019 · 2019
Later among the works it cites.
Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding
Liu, J., Shahroudy, A., Perez, M.L., Wang, G., Duan, L.Y., Chichung, A.K., 2019 · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library, pp. 8026–8037
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al., 2019 · 2019
Later among the works it cites.
An attention enhanced graph convolutional lstm network for skeleton-based action recognition
Si, C., Chen, W., Wang, W., Wang, L., Tan, T., 2019 · 2019
Later among the works it cites.
A comparative review of recent kinect-based action recognition algorithms
Wang, L., Huynh, D.Q., Koniusz, P., 2019 · 2019
Later among the works it cites.
A comprehensive survey of vision-based human action recognition methods
Zhang, H.B., Zhang, Y.X., Zhong, B., Lei, Q., Yang, L., Du, J.X., Chen, D.S., 2019 · 2019
Later among the works it cites.
End-to-end object detection with transformers, in: European Conference on Computer Vision, Springer. pp. 213–229
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S., 2020 · 2020
Closest in time.
Skeleton-based action recognition with shift graph convolutional network
Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., Lu, H., 2020 · 2020
Closest in time.
Self-attention network for skeleton-based human action recognition, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 635–644
Cho, S., Maqbool, M., Liu, F., Foroosh, H., 2020 · 2020
Closest in time.
Image captioning through image transformer, in: Proceedings of the Asian Conference on Computer Vision
He, S., Liao, W., Tavakoli, H.R., Yang, M., Rosenhahn, B., Pugeault, N., 2020 · 2020
Closest in time.
Disentangling and unifying graph convolutions for skeleton-based action recognition
Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W., 2020 · 2020
Closest in time.
Learning convolutional neural networks for graphs
Niepert, M., Ahmed, M., Kutzkov, K., 2016 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…