2019

KETO: Learning Keypoint Representations for Tool Manipulation

Qin, Zengyi, Fang, Kuan, Zhu, Yuke et al.

Understand

We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals.

An efficient and informative representation would facilitate the effectiveness and generalization of such algorithms.
For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation.
For each task, a set of task-specific keypoints is jointly predicted from 3D point clouds of the tool object by a deep neural network.

Built on

Tools and human evolution
S. L. Washburn · 1960
Earlier work this paper cites.
Animal tool behavior: The use and manufacture of tools by animals
B. B. Beck · 1980
Earlier work this paper cites.
The robonaut hand: A dexterous robot hand for space
C. Lovchik and M. A. Diftler · 1999
Earlier work this paper cites.
Cognition and tool use: Forms of engagement in human and animal use of tools
C. Baber · 2003
Earlier work this paper cites.
Distinctive image features from scale-invariant keypoints
D. G. Lowe · 2004
Earlier work this paper cites.
Learning, then compacting visual policies
S. Jodogne and J. Piater · 2005
Earlier work this paper cites.
Learning features for tracking
M. Grabner, H. Grabner, and H. Bischof · 2007
Earlier work this paper cites.
Affordance-based imitation learning in robots
M. Lopes, F. S. Melo, and L. Montesano · 2007
Earlier work this paper cites.
To afford or not to afford: A new formalization of affordances toward affordance-based robot control
E. Şahin, M. Çakmak, M. R. Doğar, E. Uğur, and G. Üçoluk · 2007
Earlier work this paper cites.
Keypoint detection and local feature matching for textured 3d face recognition
A. S. Mian, M. Bennamoun, and R. Owens · 2008
Earlier work this paper cites.
Multiple 3d object tracking for augmented reality
Y. Park, V. Lepetit, and W. Woo · 2008
Earlier work this paper cites.
3d face recognition using multiview keypoint matching
M. Mayo and E. Zhang · 2009
Earlier work this paper cites.
Real-time 3d model-based tracking using edge and keypoint features for robotic manipulation
C. Choi and H. I. Christensen · 2010
Earlier work this paper cites.
Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding
J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel · 2010
Earlier work this paper cites.
Grasping the affordances, understanding the reasoning: toward a dialectical theory of human tool use
F. Osiurak, C. Jarry, and D. Le Gall · 2010
Earlier work this paper cites.
Self-supervised cross-modal online learning of basic object affordances for developmental robotic systems
B. Ridge, D. Skočaj, and A. Leonardis · 2010
Earlier work this paper cites.
Gravity-based robotic cloth folding
J. Van Den Berg, S. Miller, K. Goldberg, and P. Abbeel · 2010
Earlier work this paper cites.
3d facial expression recognition using sift descriptors of automatically detected keypoints
S. Berretti, B. B. Amor, M. Daoudi, and A. Del Bimbo · 2011
Earlier work this paper cites.
Parametrized shape models for clothing
S. Miller, M. Fritz, T. Darrell, and P. Abbeel · 2011
Earlier work this paper cites.
Orb: An efficient alternative to sift or surf
E. Rublee, V. Rabaud, K. Konolige, and G. R. Bradski · 2011
Earlier work this paper cites.

Similar

Freak: Fast retina keypoint
A. Alahi, R. Ortiz, and P. Vandergheynst · 2012
Cited alongside, same era.
Efficient online structured output learning for keypoint-based object tracking
S. Hare, A. Saffari, and P. H. Torr · 2012
Cited alongside, same era.
Auto-encoding variational bayes
Original
D. P. Kingma and M. Welling · 2013
Cited alongside, same era.
Learning human activities and object affordances from rgb-d videos
H. S. Koppula, R. Gupta, and A. Saxena · 2013
Cited alongside, same era.
The ecological approach to visual perception: classic edition
J. J. Gibson · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
D. P. Kingma and J. Ba · 2014
Cited alongside, same era.
Consensus-based matching and tracking of keypoints for object tracking
G. Nebehay and R. Pflugfelder · 2014
Cited alongside, same era.
Deeppose: Human pose estimation via deep neural networks
A. Toshev and C. Szegedy · 2014
Cited alongside, same era.
Category-specific object reconstruction from a single image
A. Kar, S. Tulsiani, J. Carreira, and J. Malik · 2015
Cited alongside, same era.
Viewpoints and keypoints
S. Tulsiani and J. Malik · 2015
Cited alongside, same era.
Understanding tools: Task-oriented object modeling, learning and recognition
Y. Zhu, Y. Zhao, and S. Chun Zhu · 2015
Cited alongside, same era.
Learning to poke by poking: Experiential learning of intuitive physics
P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine · 2016
Cited alongside, same era.

Then

Single image 3d interpreter network
J. Wu, T. Xue, J. J. Lim, Y. Tian, J. B. Tenenbaum, A. Torralba, and W. T. Freeman · 2016
Later among the works it cites.
Recurrent human pose estimation
V. Belagiannis and A. Zisserman · 2017
Later among the works it cites.
Realtime multi-person 2d pose estimation using part affinity fields
Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh · 2017
Later among the works it cites.
Robust adaptive fusion tracking based on complex cells and keypoints
S. Chan, X. Zhou, and S. Chen · 2017
Later among the works it cites.
Deep visual foresight for planning robot motion
C. Finn and S. Levine · 2017
Later among the works it cites.
Affordance detection for task-specific grasping using deep learning
M. Kokic, J. A. Stork, J. A. Haustein, and D. Kragic · 2017
Later among the works it cites.
Pointnet: Deep learning on point sets for 3d classification and segmentation
C. R. Qi, H. Su, K. Mo, and L. J. Guibas · 2017
Later among the works it cites.
Domain randomization for transferring deep neural networks from simulation to the real world
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel · 2017
Later among the works it cites.
Using simulation and domain adaptation to improve efficiency of deep robotic grasping
K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, et al · 2018
Later among the works it cites.
Affordancenet: An end-to-end deep learning approach for object affordance detection
T.-T. Do, A. Nguyen, and I. Reid · 2018
Later among the works it cites.
Multi-task domain adaptation for deep learning of instance grasping from simulation
K. Fang, Y. Bai, S. Hinterstoisser, S. Savarese, and M. Kalakrishnan · 2018
Later among the works it cites.
Learning task-oriented grasping for tool manipulation from simulated self-supervision
K. Fang, Y. Zhu, A. Garg, A. Kuryenkov, V. Mehta, L. Fei-Fei, and S. Savarese · 2018
Later among the works it cites.
Robot bed-making: Deep transfer learning using depth sensing of deformable fabric
Original
D. Seita, N. Jamali, M. Laskey, R. Berenstein, A. K. Tanwani, P. Baskaran, S. Iba, J. Canny, and K. Goldberg · 2018
Later among the works it cites.
Differentiable physics and stable modes for tool-use and manipulation planning
M. Toussaint, K. Allen, K. A. Smith, and J. B. Tenenbaum · 2018
Later among the works it cites.
Human-guided trajectory adaptation for tool transfer
T. Fitzgerald, E. Short, A. Goel, and A. Thomaz · 2019
Closest in time.
Force-and-motion constrained planning for tool use
R. Holladay, T. Lozano-Perez, and A. Rodriguez · 2019
Closest in time.
kpam: Keypoint affordances for category-level robotic manipulation
Original
L. Manuelli, W. Gao, P. Florence, and R. Tedrake · 2019
Closest in time.
6-dof graspnet: Variational grasp generation for object manipulation
Original
A. Mousavian, C. Eppner, and D. Fox · 2019
Closest in time.
Improvisation through physical understanding: Using novel objects as tools with visual foresight
Original
A. Xie, F. Ebert, S. Levine, and C. Finn · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…