2019

Learning Temporal Pose Estimation from Sparsely-Labeled Videos

Bertasius, Gedas, Feichtenhofer, Christoph, Tran, Du et al.

Understand

Modern approaches for multi-person pose estimation in video require large amounts of dense annotations.

However, labeling every frame in a video is costly and labor intensive.
To reduce the need for dense annotations, we propose a PoseWarper network that leverages training videos with sparse annotations (every k frames) to learn to perform dense temporal pose propagation and estimation.
Given a pair of video frames---a labeled Frame A and an unlabeled Frame B---we train our model to predict human pose in Frame A using the features from Frame B by means of deformable convolutions to implicitly learn the pose warping between A and B.

Built on

Understanding neural networks through deep visualization
Original
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson · 1906
Earlier work this paper cites.
Two-frame motion estimation based on polynomial expansion
Gunnar Farnebäck · 2003
Earlier work this paper cites.
Beyond trees: common-factor models for 2d human pose recovery
Xiangyang Lan and D. P. Huttenlocher · 2005
Earlier work this paper cites.
Measure locally, reason globally: Occlusion-sensitive articulated pose estimation
L. Sigal and M. J. Black · 2006
Earlier work this paper cites.
Multiple tree models for occlusion and spatial constraints in human pose estimation
Yang Wang and Greg Mori · 2008
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Pictorial structures revisited: People detection and articulated pose estimation
Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele · 2009
Earlier work this paper cites.
Monocular 3d pose estimation and tracking by detection
M. Andriluka, S. Roth, and B. Schiele · 2010
Earlier work this paper cites.
Clustered pose and nonlinear appearance models for human pose estimation
Sam Johnson and Mark Everingham · 2010
Earlier work this paper cites.
Articulated pose estimation with flexible mixtures-of-parts
Yi Yang and D. Ramanan · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012
Earlier work this paper cites.
Strong appearance and expressive spatial models for human pose estimation
Leonid Pishchulin, Mykhaylo Andriluka, Peter V. Gehler, and Bernt Schiele · 2013
Earlier work this paper cites.
Human pose estimation using body parts dependent joint regressors
M. Dantone, J. Gall, C. Leistner, and L. van Gool · 2013
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Original
Matthew D. Zeiler and Rob Fergus · 2013
Earlier work this paper cites.
Deeppose: Human pose estimation via deep neural networks
Alexander Toshev and Christian Szegedy · 2014
Earlier work this paper cites.
Spatial pyramid pooling in deep convolutional networks for visual recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik · 2014
Earlier work this paper cites.
Learning rich features from RGB-D images for object detection and segmentation
Saurabh Gupta, Ross Girshick, Pablo Arbelaez, and Jitendra Malik · 2014
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick · 2014
Earlier work this paper cites.
Articulated pose estimation by a graphical model with image dependent pairwise relations
Xianjie Chen and Alan L Yuille · 2014
Earlier work this paper cites.

Similar

Multi-source deep learning for human pose estimation
W. Ouyang, X. Chu, and X. Wang · 2014
Cited alongside, same era.
Joint training of a convolutional network and a graphical model for human pose estimation
Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler · 2014
Cited alongside, same era.
Deepedge: A multi-scale bifurcated deep network for top-down contour detection
Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani · 2015
Cited alongside, same era.
Faster R-CNN: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun · 2015
Cited alongside, same era.
Fast R-CNN
Ross Girshick · 2015
Cited alongside, same era.
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich · 2015
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Cited alongside, same era.
Flowing convnets for human pose estimation in videos
T. Pfister, J. Charles, and A. Zisserman · 2015
Cited alongside, same era.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Cited alongside, same era.
Ssd: Single shot multibox detector
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg · 2016
Cited alongside, same era.
R-fcn: Object detection via region-based fully convolutional networks
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun · 2016
Cited alongside, same era.
You only look once: Unified, real-time object detection
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi · 2016
Cited alongside, same era.

Then

Aggregated residual transformations for deep neural networks
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He · 2017
Later among the works it cites.
Convolutional random walk networks for semantic image segmentation
Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, and Jianbo Shi · 2017
Later among the works it cites.
Mask R-CNN
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick · 2017
Later among the works it cites.
Focal Loss for Dense Object Detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár · 2017
Later among the works it cites.
YOLO9000: better, faster, stronger
Joseph Redmon and Ali Farhadi · 2017
Later among the works it cites.
Posetrack: Joint multi-person pose estimation and tracking
Umar Iqbal, Anton Milan, and Juergen Gall · 2017
Later among the works it cites.
Realtime multi-person 2d pose estimation using part affinity fields
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh · 2017
Later among the works it cites.
Deformable convolutional networks
J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei · 2017
Later among the works it cites.
Towards accurate multi-person pose estimation in the wild
George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and Kevin Murphy · 2017
Later among the works it cites.
Recurrent human pose estimation
Vasileios Belagiannis and Andrew Zisserman · 2017
Later among the works it cites.
Articulated multi-person tracking in the wild
Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres, and Bernt Schiele · 2017
Later among the works it cites.
Thin-slicing network: A deep structured model for pose estimation in videos
Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges · 2017
Later among the works it cites.
Rmpe: Regional multi-person pose estimation
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu · 2017
Later among the works it cites.
Simple baselines for human pose estimation and tracking
Bin Xiao, Haiping Wu, and Yichen Wei · 2018
Later among the works it cites.
Detect-and-Track: Efficient Pose Estimation in Videos
Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran · 2018
Later among the works it cites.
Poseflow: A deep motion representation for understanding human behaviors in videos
Dingwen Zhang, Guangyu Guo, Dong Huang, and Junwei Han · 2018
Later among the works it cites.
Self-supervised learning of a facial attribute embedding from video
O. Wiles, A.S. Koepke, and A. Zisserman · 2018
Later among the works it cites.
Multi-domain pose network for multi-person pose estimation and tracking
Hengkai Guo, Tang Tang, Guozhong Luo, Riwei Chen, Yongchen Lu, and Linfu Wen · 2018
Later among the works it cites.
Pose Flow: Efficient online pose tracking
Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu · 2018
Later among the works it cites.
Deep high-resolution representation learning for human pose estimation
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…