2018

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Lee, Michelle A., Zhu, Yuke, Srinivasan, Krishnan et al.

Understand

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback.

However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics.
While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity.
We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning.

Built on

“Quasi-Static Assembly of Compliantly Supported Rigid Parts”
D Whitney · 1982
Earlier work this paper cites.
“Neural Darwinism: The theory of neuronal group selection.”
Gerald Edelman · 1987
Earlier work this paper cites.
“Historical Perspective and State of the Art in Robot Force Control”
D.. Whitney · 1987
Earlier work this paper cites.
“Integrated tactile sensing for gripper fingers”
A Bicchi, M Bergamasco, P Dario and A Fiorillo · 1988
Earlier work this paper cites.
“Assembly strategies for chamferless parts”
M.. Caine, T. Lozano-Perez and W.. Seering · 1989
Earlier work this paper cites.
“Inertial Properties in Robotic Manipulation: An Object-Level Framework”
O. Khatib · 1995
Earlier work this paper cites.
“The CHAI libraries”
F. Conti et al · 2003
Earlier work this paper cites.
“Neural synergy between kinetic vision and touch”
Randolph Blake, Kenith Sobel and Thomas James · 2004
Earlier work this paper cites.
“Learning tactile characterizations of object- and pose-specific grasps”
Y. Bekiroglu, R. Detry and D. Kragic · 2011
Earlier work this paper cites.
“Learning force control policies for compliant manipulation”
M. Kalakrishnan, L. Righetti, P. Pastor and S. Schaal · 2011
Earlier work this paper cites.
“Multimodal deep learning”
Jiquan Ngiam et al · 2011
Earlier work this paper cites.
“Human-inspired robotic grasp control with tactile sensing”
Joseph Romano et al · 2011
Earlier work this paper cites.
“Multimodal learning with deep boltzmann machines”
Nitish Srivastava and Ruslan Salakhutdinov · 2012
Earlier work this paper cites.
“A probabilistic framework for task-oriented grasp stability assessment”
Y. Bekiroglu, D. Song, L. Wang and D. Kragic · 2013
Earlier work this paper cites.
“A direct method for trajectory optimization of rigid bodies through contact”
M Posa, C Cantu and R Tedrake · 2014
Earlier work this paper cites.
“An autonomous manipulation system based on force control and optimization”
Ludovic Righetti et al · 2014
Earlier work this paper cites.
“Learning relational object categories using behavioral exploration and multimodal perception”
Jivko Sinapov, Connor Schenck and Alexander Stoytchev · 2014
Earlier work this paper cites.
“Automated guidance of peg-in-hole assembly tasks for complex-shaped parts”
H. Song, Y. Kim and J. Song · 2014
Earlier work this paper cites.
“Adaptation of manipulation skills in physical contact with the environment to reference force profiles”
Fares Abu-Dakka et al · 2015
Earlier work this paper cites.
“Exploitation of Environmental Constraints in Human and Robotic Grasping”
Clemens Eppner et al · 2015
Earlier work this paper cites.

Similar

“FlowNet: Learning Optical Flow with Convolutional Networks”
Original
Philipp Fischer et al · 2015
Cited alongside, same era.
“Data-Driven Online Decision Making for Autonomous Manipulation”
Daniel Kappler et al · 2015
Cited alongside, same era.
“Deep learning”
Yann LeCun, Yoshua Bengio and Geoffrey Hinton · 2015
Cited alongside, same era.
“Continuous control with deep reinforcement learning”
Original
Timothy Lillicrap et al · 2015
Cited alongside, same era.
“Action-conditional video prediction using deep networks in atari games”
Junhyuk Oh et al · 2015
Cited alongside, same era.
“Trust region policy optimization”
John Schulman et al · 2015
Cited alongside, same era.
“Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor”
Z. Su et al · 2015
Cited alongside, same era.
“Learning robot in-hand manipulation with tactile features”
Herke Van, Tucker Hermans, Gerhard Neumann and Jan Peters · 2015
Cited alongside, same era.
“Stabilizing novel objects by learning to predict tactile slip”
Filipe Veiga, Herke Van, Jan Peters and Tucker Hermans · 2015
Cited alongside, same era.
“Learning to poke by poking: Experiential learning of intuitive physics”
Pulkit Agrawal et al · 2016
Cited alongside, same era.
“A framework for real-time multi-contact multi-body dynamic simulation”
Francois Conti and Oussama Khatib · 2016
Cited alongside, same era.
“One-shot learning of manipulation skills with online dynamics adaptation and neural network priors”
Justin Fu, Sergey Levine and Pieter Abbeel · 2016
Cited alongside, same era.

Then

“Interactive Perception: Leveraging Action in Perception and Perception in Action”
Jeannette Bohg et al · 2017
Later among the works it cites.
“The feeling of success: Does touch sensing help predict grasp outcomes?”
Roberto Calandra et al · 2017
Later among the works it cites.
“Path integral guided policy search”
Yevgen Chebotar et al · 2017
Later among the works it cites.
“Fundamental limitations in performance and interpretability of common planar rigid-body contact models”
Original
Nima Fazeli, Samuel Zapolsky, Evan Drumwright and Alberto Rodriguez · 2017
Later among the works it cites.
“Deep visual foresight for planning robot motion”
Chelsea Finn and Sergey Levine · 2017
Later among the works it cites.
“Probabilistic Articulated Real-Time Tracking for Robot Manipulation”
Cristina Garcia et al · 2017
Later among the works it cites.
“Learning end-to-end multimodal sensor policies for autonomous navigation”
Original
Guan-Horng Liu et al · 2017
Later among the works it cites.
“Learning to represent haptic feedback for partially-observable tasks”
Jaeyong Sung, J Salisbury and Ashutosh Saxena · 2017
Later among the works it cites.
“Deep Multimodal Representation Learning from Temporal Data”
Xitong Yang et al · 2017
Later among the works it cites.
“Learning dexterous in-hand manipulation”
Original
Marcin Andrychowicz et al · 2018
Closest in time.
“Using simulation and domain adaptation to improve efficiency of deep robotic grasping”
Konstantinos Bousmalis et al · 2018
Closest in time.
“Integrating State Representation Learning Into Deep Reinforcement Learning”
Tim de Bruin, Jens Kober, Karl Tuyls and Robert Babuska · 2018
Closest in time.
“More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch”
R. Calandra et al · 2018
Closest in time.
“Experimental studies of contact space model for multi-surface collisions in articulated rigid-body systems”
Shameek Ganguly and Oussama Khatib · 2018
Closest in time.
“State Representation Learning for Control: An Overview”
Original
Timothée Lesort, Natalia Díaz-Rodríguez, Jean-François Goudou and David Filliat · 2018
Closest in time.
“Audio-visual scene analysis with self-supervised multisensory features”
Andrew Owens and Alexei Efros · 2018
Closest in time.
“Sim-to-real transfer of robotic control with dynamics randomization”
Xue Peng, Marcin Andrychowicz, Wojciech Zaremba and Pieter Abbeel · 2018
Closest in time.
“An Efficient Acyclic Contact Planner for Multiped Robots”
S. Tonneau et al · 2018
Closest in time.
“Reinforcement and imitation learning for diverse visuomotor skills”
Original
Yuke Zhu et al · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…