2019

Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop

Kolotouros, Nikos, Pavlakos, Georgios, Black, Michael J. et al.

Understand

Model-based human pose estimation is currently approached through two different paradigms.

Optimization-based methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate image-model alignments, but are often slow and sensitive to the initialization.
In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision.
In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration.

Built on

SCAPE: shape completion and animation of people
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis · 2005
Earlier work this paper cites.
Combined discriminative and generative articulated pose and non-rigid shape estimation
Leonid Sigal, Alexandru Balan, and Michael J Black · 2008
Earlier work this paper cites.
Estimating human shape and pose from a single image
Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael J Black · 2009
Earlier work this paper cites.
Clustered pose and nonlinear appearance models for human pose estimation
Sam Johnson and Mark Everingham · 2010
Earlier work this paper cites.
Learning effective human pose estimation from inaccurate annotation
Sam Johnson and Mark Everingham · 2011
Earlier work this paper cites.
2D human pose estimation: New benchmark and state of the art analysis
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele · 2014
Earlier work this paper cites.
Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Joint training of a convolutional network and a graphical model for human pose estimation
Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler · 2014
Earlier work this paper cites.
SMPL: A skinned multi-person linear model
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black · 2015
Earlier work this paper cites.
Fully connected deep structured networks
Original
Alexander G Schwing and Raquel Urtasun · 2015
Earlier work this paper cites.
Conditional random fields as recurrent neural networks
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang, and Philip HS Torr · 2015
Earlier work this paper cites.
Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black · 2016
Earlier work this paper cites.
Deepcut: Joint subset partition and labeling for multi person pose estimation
Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter V Gehler, and Bernt Schiele · 2016
Earlier work this paper cites.
Convolutional pose machines
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh · 2016
Earlier work this paper cites.
Sparseness meets deepness: 3D human pose estimation from monocular video
Xiaowei Zhou, Menglong Zhu, Spyridon Leonardos, Konstantinos G Derpanis, and Kostas Daniilidis · 2016
Earlier work this paper cites.

Similar

Realtime multi-person 2D pose estimation using part affinity fields
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh · 2017
Cited alongside, same era.
Towards accurate marker-less human shape and pose estimation over time
Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Javier Romero, Ijaz Akhter, and Michael J. Black · 2017
Cited alongside, same era.
Unite the people: Closing the loop between 3D and 2D human representations
Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J Black, and Peter V Gehler · 2017
Cited alongside, same era.
A simple yet effective baseline for 3D human pose estimation
Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little · 2017
Cited alongside, same era.
Monocular 3D human pose estimation in the wild using improved cnn supervision
Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt · 2017
Cited alongside, same era.
VNect: Real-time 3D human pose estimation with a single RGB camera
Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt · 2017
Cited alongside, same era.
Hand keypoint detection in single images using multiview bootstrapping
Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh · 2017
Cited alongside, same era.
Indirect deep structured learning for 3D human body shape and pose prediction
Vince Tan, Ignas Budvytis, and Roberto Cipolla · 2017
Cited alongside, same era.
Learning to fuse 2D and 3D image cues for monocular body pose estimation
Bugra Tekin, Pablo Márquez-Neila, Mathieu Salzmann, and Pascal Fua · 2017
Cited alongside, same era.
Self-supervised learning of motion capture
Hsiao-Yu Tung, Hsiao-Wei Tung, Ersin Yumer, and Katerina Fragkiadaki · 2017
Cited alongside, same era.
OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields
Original
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh · 2018
Cited alongside, same era.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille · 2018
Cited alongside, same era.

Then

Raynet: Learning volumetric 3D reconstruction with ray potentials
Despoina Paschalidou, Osman Ulusoy, Carolin Schmitt, Luc Van Gool, and Andreas Geiger · 2018
Later among the works it cites.
Learning to estimate 3D human pose and shape from a single color image
Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis · 2018
Later among the works it cites.
Integral human pose regression
Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei · 2018
Later among the works it cites.
BodyNet: Volumetric inference of 3D human body shapes
Gul Varol, Duygu Ceylan, Bryan Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, and Cordelia Schmid · 2018
Later among the works it cites.
Recovering accurate 3D human pose in the wild using imus and a moving camera
Timo von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll · 2018
Later among the works it cites.
Monocular 3D pose and shape estimation of multiple people in natural scenes-the importance of multiple scene constraints
Andrei Zanfir, Elisabeta Marinoiu, and Cristian Sminchisescu · 2018
Later among the works it cites.
Deep network for the integrated 3D sensing of multiple people in natural images
Andrei Zanfir, Elisabeta Marinoiu, Mihai Zanfir, Alin-Ionut Popa, and Cristian Sminchisescu · 2018
Later among the works it cites.
Exploiting temporal context for 3D human pose estimation in the wild
Anurag Arnab, Carl Doersch, and Andrew Zisserman · 2019
Closest in time.
HoloPose: Holistic 3D human reconstruction in-the-wild
Riza Alp Guler and Iasonas Kokkinos · 2019
Closest in time.
Learning 3D human dynamics from video
Angjoo Kanazawa, Jason Zhang, Panna Felsen, and Jitendra Malik · 2019
Closest in time.
Convolutional mesh regression for single-image human shape reconstruction
Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis · 2019
Closest in time.
Expressive body capture: 3D hands, face, and body from a single image
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black · 2019
Closest in time.
LCR-Net++: Multi-person 2D and 3D pose detection in natural images
Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid · 2019
Closest in time.
Monocular total capture: Posing face, body, and hands in the wild
Donglai Xiang, Hanbyul Joo, and Yaser Sheikh · 2019
Closest in time.
On the continuity of rotation representations in neural networks
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…