2020

Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action Recognition

Shi, Lei, Zhang, Yifan, Cheng, Jian et al.

Understand

Dynamic skeletal data, represented as the 2D/3D coordinates of human joints, has been widely studied for human action recognition due to its high-level semantic information and environmental robustness.

However, previous methods heavily rely on designing hand-crafted traversal rules or graph topologies to draw dependencies between the joints, which are limited in performance and generalizability.
In this work, we present a novel decoupled spatial-temporal attention network(DSTA-Net) for skeleton-based action recognition.
It involves solely the attention blocks, allowing for modeling spatial-temporal dependencies between joints without the requirement of knowing their positions or mutual connections.

Built on

Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML. vol. 30, p. 3 (2013)
2013
Earlier work this paper cites.
De Smedt, Q., Wannous, H., Vandeborre, J.P.: Skeleton-Based Dynamic Hand Gesture Recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1206–1214 (Jun 2016)
2016
Earlier work this paper cites.
Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: A Large Scale Dataset for 3d Human Activity Analysis. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1010–1019 (2016)
2016
Earlier work this paper cites.
Carreira, J., Zisserman, A.: Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6299–6308 (Jul 2017)
2017
Earlier work this paper cites.
De Smedt, Q., Wannous, H., Vandeborre, J.P., Guerry, J., Le Saux, B., Filliat, D.: SHREC’17 Track: 3d Hand Gesture Recognition Using a Depth and Skeletal Dataset. In: Pratikakis, I., Dupont, F., Ovsjanikov, M. (eds.) Eurographics Workshop on 3D Object Retrieval. pp. 1–6 (2017)
2017
Earlier work this paper cites.
Li, C., Zhong, Q., Xie, D., Pu, S.: Skeleton-based Action Recognition with Convolutional Neural Networks. In: IEEE International Conference on Multimedia & Expo Workshops (ICMEW). pp. 597–600 (2017)
2017
Earlier work this paper cites.
Liu, J., Wang, G., Duan, L.Y., Abdiyeva, K., Kot, A.C.: Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Transactions on Image Processing 27
2017
Earlier work this paper cites.
Liu, M., Liu, H., Chen, C.: Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition 68
2017
Earlier work this paper cites.
Qiu, Z., Yao, T., Mei, T.: Learning Spatio-Temporal Representation with Pseudo-3d Residual Networks. In: The IEEE International Conference on Computer Vision (ICCV). pp. 5533–5541 (Oct 2017)
2017
Earlier work this paper cites.
Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia, P., Lillicrap, T.: A simple neural network module for relational reasoning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. pp. 4974–4983 (2017)
2017
Earlier work this paper cites.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is All you Need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. pp. 6000–6010 (2017)
2017
Earlier work this paper cites.
Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., Zheng, N.: View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition From Skeleton Data. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2117–2126 (2017)
2017
Earlier work this paper cites.

Similar

Cao, C., Lan, C., Zhang, Y., Zeng, W., Lu, H., Zhang, Y.: Skeleton-Based Action Recognition with Gated Convolutional Neural Networks. IEEE Transactions on Circuits and Systems for Video Technology pp. 3247–3257 (2018)
2018
Cited alongside, same era.
Hou, J., Wang, G., Chen, X., Xue, J.H., Zhu, R., Yang, H.: Spatial-temporal attention res-TCN for skeleton-based dynamic hand gesture recognition. In: The European Conference on Computer Vision (ECCV). pp. 0–0 (2018)
2018
Cited alongside, same era.
Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation Networks for Object Detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2018)
2018
Cited alongside, same era.
Li, S., Li, W., Cook, C., Zhu, C., Gao, Y.: Independently recurrent neural network (indrnn): Building A longer and deeper RNN. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5457–5466 (2018)
2018
Cited alongside, same era.
Liu, M., Yuan, J.: Recognizing Human Actions as the Evolution of Pose Estimation Maps. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1159–1168 (2018)
2018
Cited alongside, same era.
Si, C., Jing, Y., Wang, W., Wang, L., Tan, T.: Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning. In: The European Conference on Computer Vision (ECCV). pp. 103–118 (Sep 2018)
2018
Cited alongside, same era.
Tang, Y., Tian, Y., Lu, J., Li, P., Zhou, J.: Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5323–5332 (2018)
2018
Cited alongside, same era.
Wang, X., Girshick, R., Gupta, A., He, K.: Non-Local Neural Networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Jun 2018)
2018
Cited alongside, same era.
Yan, S., Xiong, Y., Lin, D.: Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. In: AAAI (2018)
2018
Cited alongside, same era.
Caetano, C., Sena, J., Brémond, F., Dos Santos, J.A., Schwartz, W.R.: SkeleMotion: A New Representation of Skeleton Joint Sequences based on Motion Information for 3d Action Recognition. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). pp. 1–8 (Sep 2019)
2019
Cited alongside, same era.
Chen, Y., Zhao, L., Peng, X., Yuan, J., Metaxas, D.N.: Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention. In: BMVC (2019)
2019
Cited alongside, same era.

Then

Dai, Z., Yang, Z., Yang, Y., Cohen, W.W., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. arXiv:1901.02860 (2019)
Original
2019
Later among the works it cites.
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 6202–6211 (2019)
2019
Later among the works it cites.
Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3146–3154 (2019)
2019
Later among the works it cites.
Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.Y., Kot, A.C.: NTU RGB+D 120: A Large-Scale Benchmark for 3d Human Activity Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1–1 (2019)
2019
Later among the works it cites.
Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Apr 2019)
2019
Later among the works it cites.
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Action Recognition via Pose-Based Graph Convolutional Networks with Intermediate Dense Supervision. arXiv:1911.12509 (Nov 2019)
Original
2019
Later among the works it cites.
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-Based Action Recognition With Directed Graph Neural Networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7912–7921 (Jun 2019)
2019
Later among the works it cites.
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12026–12035 (2019)
2019
Later among the works it cites.
Shi, L., Zhang, Y., Jian, C., Hanqing, L.: Gesture Recognition using Spatiotemporal Deformable Convolutional Representation. In: IEEE International Conference on Image Processing (ICIP) (Oct 2019)
2019
Later among the works it cites.
Si, C., Chen, W., Wang, W., Wang, L., Tan, T.: An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1227–1236 (2019)
2019
Later among the works it cites.
Peng, W., Hong, X., Chen, H., Zhao, G.: Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching. In: AAAI (2020)
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…