2019

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Mandlekar, Ajay, Ramos, Fabio, Boots, Byron et al.

Understand

Learning from offline task demonstrations is a problem of great interest in robotics.

For simple short-horizon manipulation tasks with modest variation in task instances, offline learning from a small set of demonstrations can produce controllers that successfully solve the task.
However, leveraging a fixed batch of data can be problematic for larger datasets and longer-horizon tasks with greater variations.
The data can exhibit substantial diversity and consist of suboptimal solution approaches.

Built on

“Alvinn: An autonomous land vehicle in a neural network”
Dean Pomerleau · 1989
Earlier work this paper cites.
“Imagenet: A large-scale hierarchical image database”
Jia Deng et al · 2009
Earlier work this paper cites.
“The columbia grasp database”
Corey Goldfeder, Matei Ciocarlie, Hao Dang and Peter Allen · 2009
Earlier work this paper cites.
“Inverse reinforcement learning”
Pieter Abbeel and Andrew Ng · 2011
Earlier work this paper cites.
“The KIT object models database: An object model database for object recognition, localization and manipulation in service robotics”
Alexander Kasper, Zhixing Xue and Rüdiger Dillmann · 2012
Earlier work this paper cites.
“Auto-encoding variational bayes”
Original
Diederik Kingma and Max Welling · 2013
Earlier work this paper cites.
“Learning monocular reactive uav control in cluttered natural environments”
Stéphane Ross et al · 2013
Earlier work this paper cites.
“Reinforcement and imitation learning via interactive no-regret learning”
Original
Stephane Ross and J Bagnell · 2014
Earlier work this paper cites.
“Learning hand-eye coordination for robotic grasping with large-scale data collection”
Sergey Levine, Peter Pastor, Alex Krizhevsky and Deirdre Quillen · 2016
Earlier work this paper cites.
“Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours”
Lerrel Pinto and Abhinav Gupta · 2016
Earlier work this paper cites.
“Learning from demonstrations through the use of non-rigid registration”
John Schulman, Jonathan Ho, Cameron Lee and Pieter Abbeel · 2016
Earlier work this paper cites.
“More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing”
Kuan-Ting Yu, Maria Bauza, Nima Fazeli and Alberto Rodriguez · 2016
Earlier work this paper cites.
“Hindsight experience replay”
Marcin Andrychowicz et al · 2017
Earlier work this paper cites.

Similar

“beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.”
Irina Higgins et al · 2017
Cited alongside, same era.
“Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics”
Original
Jeffrey Mahler et al · 2017
Cited alongside, same era.
“Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards”
Original
Matej Večerík et al · 2017
Cited alongside, same era.
“SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark”
Linxi Fan et al · 2018
Cited alongside, same era.
“Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision”
Kuan Fang et al · 2018
Cited alongside, same era.
“Off-Policy Deep Reinforcement Learning without Exploration”
Original
Scott Fujimoto, David Meger and Doina Precup · 2018
Cited alongside, same era.
“Reinforcement learning from imperfect demonstrations”
Original
Yang Gao et al · 2018
Cited alongside, same era.
“Deep q-learning from demonstrations”
Todd Hester et al · 2018
Cited alongside, same era.
“Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation”
Original
Dmitry Kalashnikov et al · 2018
Cited alongside, same era.
“RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation”
Ajay Mandlekar et al · 2018
Cited alongside, same era.
“Data-efficient hierarchical reinforcement learning”
Ofir Nachum, Shixiang Gu, Honglak Lee and Sergey Levine · 2018
Cited alongside, same era.
“Overcoming exploration in reinforcement learning with demonstrations”
Ashvin Nair et al · 2018
Cited alongside, same era.

Then

“Know What You Don’t Know: Unanswerable Questions for SQuAD”
Original
Pranav Rajpurkar, Robin Jia and Percy Liang · 2018
Later among the works it cites.
“Reinforcement and imitation learning for diverse visuomotor skills”
Yuke Zhu et al · 2018
Later among the works it cites.
“Towards Characterizing Divergence in Deep Q-Learning”
Original
Joshua Achiam, Ethan Knight and Pieter Abbeel · 2019
Closest in time.
“Striving for Simplicity in Off-policy Deep Reinforcement Learning”
Original
Rishabh Agarwal, Dale Schuurmans and Mohammad Norouzi · 2019
Closest in time.
“CrossNorm: Normalization for Off-Policy TD Reinforcement Learning”
Original
Aditya Bhatt, Max Argus, Artemij Amiranashvili and Thomas Brox · 2019
Closest in time.
“Goal-conditioned Imitation Learning”
Original
Yiming Ding, Carlos Florensa, Mariano Phielipp and Pieter Abbeel · 2019
Closest in time.
“Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog”
Original
Natasha Jaques et al · 2019
Closest in time.
“SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards”
Sanjay Krishnan et al · 2019
Closest in time.
“Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction”
Original
Aviral Kumar, Justin Fu, George Tucker and Sergey Levine · 2019
Closest in time.
“Off-Policy Policy Gradient with State Distribution Correction”
Original
Yao Liu, Adith Swaminathan, Alekh Agarwal and Emma Brunskill · 2019
Closest in time.
“Learning Latent Plans from Play”
Original
Corey Lynch et al · 2019
Closest in time.
“Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos”
Original
Aysegul Dundar et al · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…