2020

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition

Kalfaoglu, M. Esat, Kalkan, Sinan, Alatan, A. Aydin

Understand

In this work, we combine 3D convolution with late temporal modeling for action recognition.

For this aim, we replace the conventional Temporal Global Average Pooling (TGAP) layer at the end of 3D convolutional architecture with the Bidirectional Encoder Representations from Transformers (BERT) layer in order to better utilize the temporal information with BERT's attention mechanism.
We show that this replacement improves the performances of many popular 3D convolution architectures for action recognition, including ResNeXt, I3D, SlowFast and R(2+1)D.
Moreover, we provide the-state-of-the-art results on both HMDB51 and UCF101 datasets with 85.10% and 98.69% top-1 accuracy, respectively.

Built on

Tran, D., Wang, H., Feiszli, M., Torresani, L.: Video classification with channel-separated convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision. vol. 2019-Octob, pp. 5551–5560. Institute of Electrical and Electronics Engineers Inc. (4 2019). https://doi.org/10.1109/ICCV.2019.00565, http://arxiv.org/abs/1904.02811
Original
1904
Earlier work this paper cites.
Ghadiyaram, D., Feiszli, M., Tran, D., Yan, X., Wang, H., Mahajan, D.: Large-scale weakly-supervised pre-training for video action recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June
Original
1905
Earlier work this paper cites.
Wang, L., Koniusz, P., Huynh, D.Q.: Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs. Proceedings of the IEEE International Conference on Computer Vision 2019-October
Original
1906
Earlier work this paper cites.
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: A large video database for human motion recognition. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2556–2563 (2011). https://doi.org/10.1109/ICCV.2011.6126543
2011
Earlier work this paper cites.
Soomro, K., Zamir, A.R., Shah, M.: UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild (12 2012), http://arxiv.org/abs/1212.0402
Original
2012
Earlier work this paper cites.
Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision (2013). https://doi.org/10.1109/ICCV.2013.441
2013
Earlier work this paper cites.
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Li, F.F.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2014). https://doi.org/10.1109/CVPR.2014.223
2014
Earlier work this paper cites.
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems. vol. 1, pp. 568–576. Neural information processing systems foundation (2014)
2014
Earlier work this paper cites.
Ng, J.Y.H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: Deep networks for video classification. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 07-12-June, pp. 4694–4702. IEEE Computer Society (10 2015). https://doi.org/10.1109/CVPR.2015.7299101
2015
Earlier work this paper cites.
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision (2015). https://doi.org/10.1109/ICCV.2015.510
2015
Earlier work this paper cites.
Wang, L., Xiong, Y., Wang, Z., Qiao, Y.: Towards Good Practices for Very Deep Two-Stream ConvNets (7 2015), http://arxiv.org/abs/1507.02159
Original
2015
Earlier work this paper cites.
Donahue, J., Hendricks, L.A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., Darrell, T.: Long-Term Recurrent Convolutional Networks for Visual Recognition and Description. IEEE Transactions on Pattern Analysis and Machine Intelligence 39
2016
Earlier work this paper cites.

Similar

Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional Two-Stream Network Fusion for Video Action Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2016-Decem, pp. 1933–1941. IEEE Computer Society (12 2016). https://doi.org/10.1109/CVPR.2016.213
2016
Cited alongside, same era.
Hendrycks, D., Gimpel, K.: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR abs/1606.08415
Original
2016
Cited alongside, same era.
Carreira, J., Zisserman, A.: Quo Vadis, action recognition? A new model and the kinetics dataset. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. vol. 2017-Janua, pp. 4724–4733. Institute of Electrical and Electronics Engineers Inc. (11 2017). https://doi.org/10.1109/CVPR.2017.502
2017
Cited alongside, same era.
Du, W., Wang, Y., Qiao, Y.: Recurrent spatial-temporal attention network for action recognition in videos. IEEE Transactions on Image Processing 27
2017
Cited alongside, same era.
Girdhar, R., Ramanan, D., Gupta, A., Sivic, J., Russell, B.: ActionVLAD: Learning spatio-temporal aggregation for action classification. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (2017). https://doi.org/10.1109/CVPR.2017.337
2017
Cited alongside, same era.
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (4 2017), http://arxiv.org/abs/1704.04861
Original
2017
Cited alongside, same era.
Li, Z., Gavrilyuk, K., Gavves, E., Jain, M., Snoek, C.G.: VideoLSTM convolves, attends and flows for action recognition. Computer Vision and Image Understanding 166
2017
Cited alongside, same era.
Loshchilov, I., Hutter, F.: Decoupled Weight Decay Regularization. 7th International Conference on Learning Representations, ICLR 2019 (11 2017), http://arxiv.org/abs/1711.05101
Original
2017
Cited alongside, same era.
Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems pp. 4905–4913 (1 2017), http://arxiv.org/abs/1701.04128
Original
2017
Cited alongside, same era.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Å., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 2017-Decem, pp. 5999–6009. Neural information processing systems foundation (2017)
2017
Cited alongside, same era.
Chen, Y., Kalantidis, Y., Li, J., Yan, S., Feng, J.: Multi-fiber Networks for Video Recognition. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 11205 LNCS, pp. 364–380. Springer Verlag (7 2018). https://doi.org/10.1007/978-3-030-01246-5_22
2018
Cited alongside, same era.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (10 2018), http://arxiv.org/abs/1810.04805
Original
2018
Cited alongside, same era.

Then

Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video Action Transformer Network. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June
Original
2018
Later among the works it cites.
Hara, K., Kataoka, H., Satoh, Y.: Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 6546–6555. IEEE Computer Society (12 2018). https://doi.org/10.1109/CVPR.2018.00685
2018
Later among the works it cites.
Lin, J., Gan, C., Han, S.: TSM: Temporal Shift Module for Efficient Video Understanding. Proceedings of the IEEE International Conference on Computer Vision 2019-October
Original
2018
Later among the works it cites.
Piergiovanni, A., Angelova, A., Toshev, A., Ryoo, M.S.: Evolving Space-Time Neural Architectures for Videos. Proceedings of the IEEE International Conference on Computer Vision 2019-October
Original
2018
Later among the works it cites.
Piergiovanni, A., Ryoo, M.S.: Representation Flow for Action Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June
Original
2018
Later among the works it cites.
Tran, D., Wang, H., Torresani, L., Ray, J., Lecun, Y., Paluri, M.: A Closer Look at Spatiotemporal Convolutions for Action Recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 6450–6459. IEEE Computer Society (12 2018). https://doi.org/10.1109/CVPR.2018.00675
2018
Later among the works it cites.
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal Segment Networks for Action Recognition in Videos (2018). https://doi.org/10.1109/TPAMI.2018.2868668
2018
Later among the works it cites.
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local Neural Networks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 7794–7803. IEEE Computer Society (12 2018). https://doi.org/10.1109/CVPR.2018.00813
2018
Later among the works it cites.
Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 11219 LNCS, pp. 318–335. Springer Verlag (2018). https://doi.org/10.1007/978-3-030-01267-0_19
2018
Later among the works it cites.
Crasto, N., Weinzaepfel, P., Alahari, K., Schmid, C.: MARS: Motion-augmented rgb stream for action recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2019-June, pp. 7874–7883. IEEE Computer Society (6 2019). https://doi.org/10.1109/CVPR.2019.00807
2019
Later among the works it cites.
Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition. In: Proceedings of the IEEE International Conference on Computer Vision. vol. 2019-October, pp. 6201–6210. Institute of Electrical and Electronics Engineers Inc. (10 2019). https://doi.org/10.1109/ICCV.2019.00630
2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…