2022

Universal Prototype Transport for Zero-Shot Action Recognition and Localization

Mettes, Pascal

Understand

This work addresses the problem of recognizing action categories in videos when no training examples are available.

The current state-of-the-art enables such a zero-shot recognition by learning universal mappings from videos to a semantic space, either trained on large-scale seen actions or on objects.
While effective, we find that universal action and object mappings are biased to specific regions in the semantic space.
These biases lead to a fundamental problem: many unseen action categories are simply never inferred during testing.

Built on

Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1933–1941
1941
Earlier work this paper cites.
Kantorovich LV (1942) On the translocation of masses. In: Dokl. Akad. Nauk. USSR (NS), vol 37, pp 199–201
1942
Earlier work this paper cites.
Mettes P, Thong W, Snoek CGM (2021) Object priors for classifying and localizing unseen actions. International Journal of Computer Vision 129:1954–1971
1971
Earlier work this paper cites.
Banerjee A, Dhillon IS, Ghosh J, Sra S, Ridgeway G (2005) Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research 6(9)
2005
Earlier work this paper cites.
Vapnik V (2006) Estimation of dependences based on empirical data. Springer Science & Business Media
2006
Earlier work this paper cites.
Arnold A, Nallapati R, Cohen WW (2007) A comparative study of methods for transductive transfer learning. In: Seventh IEEE international conference on data mining workshops (ICDMW 2007), IEEE, pp 77–82
2007
Earlier work this paper cites.
Bonneel N, Van De Panne M, Paris S, Heidrich W (2011) Displacement interpolation using lagrangian mass transport. In: Proceedings of the 2011 SIGGRAPH Asia conference, pp 1–12
2011
Earlier work this paper cites.
Liu J, Kuipers B, Savarese S (2011) Recognizing human actions by attributes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3337–3344
2011
Earlier work this paper cites.
Rohrbach M, Ebert S, Schiele B (2013) Transfer learning in a transductive setting. Advances in neural information processing systems 26
2013
Earlier work this paper cites.
Fu Y, Hospedales TM, Xiang T, Fu Z, Gong S (2014) Transductive multi-view embedding for zero-shot recognition and annotation. In: European Conference on Computer Vision, Springer, pp 584–599
2014
Earlier work this paper cites.
Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 1725–1732
2014
Earlier work this paper cites.
Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, vol 27
2014
Earlier work this paper cites.
Caba Heilbron F, Escorcia V, Ghanem B, Carlos Niebles J (2015) Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the ieee conference on computer vision and pattern recognition, pp 961–970
2015
Earlier work this paper cites.
Jain M, Van Gemert JC, Mensink T, Snoek CG (2015) Objects2action: Classifying and localizing actions without any video example. In: Proceedings of the IEEE international conference on computer vision, pp 4588–4596
2015
Earlier work this paper cites.
Kodirov E, Xiang T, Fu Z, Gong S (2015) Unsupervised domain adaptation for zero-shot learning. In: Proceedings of the IEEE international conference on computer vision, pp 2452–2460
2015
Earlier work this paper cites.
Zhang Z, Wang C, Xiao B, Zhou W, Liu S (2015) Robust relative attributes for human action recognition. Pattern Analysis and Applications 18:157–171
2015
Earlier work this paper cites.
Alexiou I, Xiang T, Gong S (2016) Exploring synonyms as context in zero-shot action recognition. In: 2016 IEEE International Conference on Image Processing (ICIP), IEEE, pp 4190–4194
2016
Earlier work this paper cites.
Li Y, Hu Sh, Li B (2016) Recognizing unseen actions in a domain-adapted embedding space. In: 2016 IEEE International Conference on Image Processing (ICIP), IEEE, pp 4195–4199
2016
Earlier work this paper cites.
Wu Z, Fu Y, Jiang YG, Sigal L (2016) Harnessing object and scene semantics for large-scale video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3112–3121
2016
Earlier work this paper cites.
Carreira J, Zisserman A (2017) Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 6299–6308
2017
Earlier work this paper cites.

Similar

Mettes P, Snoek CGM (2017) Spatial-aware object embeddings for zero-shot localization and classification of actions. In: Proceedings of the IEEE international conference on computer vision, pp 4443–4452
2017
Cited alongside, same era.
Soomro K, Shah M (2017) Unsupervised action discovery and localization in videos. In: Proceedings of the IEEE International Conference on Computer Vision, pp 696–705
2017
Cited alongside, same era.
Xu X, Hospedales T, Gong S (2017) Transductive zero-shot action recognition by word-vector embedding. International Journal of Computer Vision 123:309–333
2017
Cited alongside, same era.
Damen D, Doughty H, Farinella GM, Fidler S, Furnari A, Kazakos E, Moltisanti D, Munro J, Perrett T, Price W, et al (2018) Scaling egocentric vision: The epic-kitchens dataset. In: European Conference on Computer Vision, pp 720–736
2018
Cited alongside, same era.
Roitberg A, Martinez M, Haurilet M, Stiefelhagen R (2018) Towards a fair evaluation of zero-shot action recognition using external data. In: European Conference on Computer Vision workshops
2018
Cited alongside, same era.
Tran D, Wang H, Torresani L, Ray J, LeCun Y, Paluri M (2018) A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 6450–6459
2018
Cited alongside, same era.
Zhu Y, Long Y, Guan Y, Newsam S, Shao L (2018) Towards universal representation for unseen action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9436–9445
2018
Cited alongside, same era.
Bishay M, Zoumpourlis G, Patras I (2019) Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition. In: British Machine Vision Conference
2019
Cited alongside, same era.
Cui Y, Jia M, Lin TY, Song Y, Belongie S (2019) Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9268–9277
2019
Cited alongside, same era.
Feichtenhofer C, Fan H, Malik J, He K (2019) Slowfast networks for video recognition. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6202–6211
2019
Cited alongside, same era.
Gao J, Zhang T, Xu C (2019) I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs. In: Proceedings of the AAAI conference on artificial intelligence, vol 33
2019
Cited alongside, same era.
Hahn M, Silva A, Rehg JM (2019) Action2vec: A crossmodal embedding approach to action learning. In: Computer Vision and Pattern Recognition workshops
2019
Cited alongside, same era.

Then

Lou A, Katsman I, Jiang Q, Belongie S, Lim SN, De Sa C (2020) Differentiating through the fréchet mean. In: International Conference on Machine Learning, PMLR, pp 6393–6403
2020
Later among the works it cites.
Mettes P, Koelma DC, Snoek CG (2020) Shuffled imagenet banks for video event detection and search. ACM Transactions on Multimedia Computing, Communications, and Applications 16(2):1–21
2020
Later among the works it cites.
Miolane N, Guigui N, Le Brigant A, Mathe J, Hou B, Thanwerdas Y, Heyder S, Peltre O, Koep N, Zaatiti H, et al (2020) Geomstats: a python package for riemannian geometry in machine learning. The Journal of Machine Learning Research 21(1):9203–9211
2020
Later among the works it cites.
Mishra A, Pandey A, Murthy HA (2020) Zero-shot learning for action recognition using synthesized features. Neurocomputing 390:117–130
2020
Later among the works it cites.
Xu Y, Han C, Qin J, Xu X, Han G, He S (2020) Transductive zero-shot action recognition via visually connected graph convolutional networks. IEEE Transactions on Neural Networks and Learning Systems 32(8):3761–3769
2020
Later among the works it cites.
Zhang L, Chang X, Liu J, Luo M, Wang S, Ge Z, Hauptmann A (2020) Zstad: Zero-shot temporal activity detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 879–888
2020
Later among the works it cites.
Arnab A, Dehghani M, Heigold G, Sun C, Lučić M, Schmid C (2021) Vivit: A video vision transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6836–6846
2021
Later among the works it cites.
Bretti C, Mettes P (2021) Zero-shot action recognition from diverse object-scene compositions. British Machine Vision Conference
2021
Later among the works it cites.
Chen S, Huang D (2021) Elaborative rehearsal for zero-shot action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 13638–13647
2021
Later among the works it cites.
Flamary R, Courty N, Gramfort A, Alaya MZ, Boisbunon A, Chambon S, Chapel L, Corenflos A, Fatras K, Fournier N, et al (2021) Pot: Python optimal transport. The Journal of Machine Learning Research 22(1):3571–3578
2021
Later among the works it cites.
Kerrigan A, Duarte K, Rawat Y, Shah M (2021) Reformulating zero-shot action recognition for multi-label actions. Advances in Neural Information Processing Systems 34:25566–25577
2021
Later among the works it cites.
Kim TS, Jones J, Peven M, Xiao Z, Bai J, Zhang Y, Qiu W, Yuille A, Hager GD (2021) Daszl: Dynamic action signatures for zero-shot learning. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 1817–1826
2021
Later among the works it cites.
Wang W, Xu H, Wang G, Wang W, Carin L (2021) Zero-shot recognition via optimal transport. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp 3471–3481
2021
Later among the works it cites.
Bao W, Yu Q, Kong Y (2022) Opental: Towards open set temporal action localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2979–2989
2022
Closest in time.
Grauman K, Westbury A, Byrne E, Chavis Z, Furnari A, Girdhar R, Hamburger J, Jiang H, Liu M, Liu X, et al (2022) Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 18995–19012
2022
Closest in time.
Lin CC, Lin K, Wang L, Liu Z, Li L (2022) Cross-modal representation learning for zero-shot action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 19978–19988
2022
Closest in time.
Pu S, Zhao K, Zheng M (2022) Alignment-uniformity aware representation learning for zero-shot video classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 19968–19977
2022
Closest in time.
Yang P, Asano YM, Mettes P, Snoek CGM (2022) Less than few: Self-shot video instance segmentation. In: European Conference on Computer Vision, Springer, pp 449–466
2022
Closest in time.
Zhang L, Chang X, Liu J, Luo M, Li Z, Yao L, Hauptmann A (2022) Tn-zstad: Transferable network for zero-shot temporal activity detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(3):3848–3861
2022
Closest in time.
Zhuo J, Zhu Y, Cui S, Wang S, MA B, Huang Q, Wei X, Wei X (2022) Zero-shot video classification with appropriate web and task knowledge transfer. In: Proceedings of the 30th ACM International Conference on Multimedia, pp 5761–5772
Original
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…