2020

MediaPipe Hands: On-device Real-time Hand Tracking

Zhang, Fan, Bazarevsky, Valentin, Vakunov, Andrey et al.

Understand

We present a real-time on-device hand tracking pipeline that predicts hand skeleton from single RGB camera for AR/VR applications.

The pipeline consists of two models: 1) a palm detector, 2) a hand landmark model.
It's implemented via MediaPipe, a framework for building cross-platform ML solutions.
The proposed model and pipeline architecture demonstrates real-time inference speed on mobile GPUs and high prediction quality.

SSD: single shot multibox detector
Original
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg · 2015
Earlier work this paper cites.
Robust articulated-icp for real-time hand tracking
Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly · 2015
Earlier work this paper cites.
Robust 3d hand pose estimation in single depth images: from single-view cnn to multi-view cnns
Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann · 2016
Earlier work this paper cites.
Feature pyramid networks for object detection
Original
Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie · 2016
Earlier work this paper cites.
Focal loss for dense object detection
Original
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár · 2017
Earlier work this paper cites.

Hand keypoint detection in single images using multiview bootstrapping
Original
Tomas Simon, Hanbyul Joo, Iain A. Matthews, and Yaser Sheikh · 2017
Cited alongside, same era.
Robust 3d hand pose estimation from single depth images using multi-view cnns
Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann · 2018
Cited alongside, same era.
Blazeface: Sub-millisecond neural face detection on mobile gpus
Original
Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann · 2019
Cited alongside, same era.
3d hand shape and pose estimation from a single rgb image
Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, and Junsong Yuan · 2019
Cited alongside, same era.
Oculus Quest Hand Tracking
Facebook
Cited in the paper.
Tensorflow lite on GPU
Google
Cited in the paper.
Efficient model-based 3d tracking of hand articulations using kinect
Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A Argyros
Cited in the paper.
Lens Studio by Snap Inc
Snapchat
Cited in the paper.

Real-time facial surface geometry from monocular video on mobile gpus
Original
Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann · 2019
Later among the works it cites.
Mediapipe: A framework for building perception pipelines
Original
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann · 2019
Later among the works it cites.
Self-supervised 3d hand pose estimation through training by fitting
Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao · 2019
Later among the works it cites.
Tensorflow.js Handpose
Google · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…