2020

Learning Robot Skills with Temporal Variational Inference

Shankar, Tanmay, Gupta, Abhinav

Understand

In this paper, we address the discovery of robotic options from demonstrations in an unsupervised manner.

Specifically, we present a framework to jointly learn low-level control policies and higher-level policies of how to use them from demonstrations of a robot performing various tasks.
By representing options as continuous latent variables, we frame the problem of learning these options as latent variable inference.
We then present a temporal formulation of variational inference based on a temporal factorization of trajectory likelihoods,that allows us to infer options in an unsupervised manner.

Built on

Strips: A new approach to the application of theorem proving to problem solving
Fikes, R. E. and Nilsson, N. J · 1971
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J · 1992
Earlier work this paper cites.
Behavioural cloning in control of a dynamic system
Esmaili, N., Sammut, C., and Shirazi, G · 1995
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
Learning from demonstration
Schaal, S · 1997
Earlier work this paper cites.
Directed information for channels with feedback
Kramer, G · 1998
Earlier work this paper cites.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., and Singh, S · 1999
Earlier work this paper cites.
Cmu graphics lab motion capture database
CMU · 2002
Earlier work this paper cites.
Visualizing high-dimensional data using t-sne
van der Maaten, L. and Hinton, G · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., and Browning, B · 2009
Earlier work this paper cites.
Learning motor primitives for robotics
Kober, J. and Peters, J · 2009
Earlier work this paper cites.
Skill chaining: Skill discovery in continuous domains
Konidaris, G. and Barto, A · 2009
Earlier work this paper cites.
Learning table tennis with a mixture of motor primitives
Muelling, K., Kober, J., and Peters, J · 2010
Earlier work this paper cites.
Movement segmentation using a primitive library
Meier, F., Theodorou, E., Stulp, F., and Schaal, S · 2011
Earlier work this paper cites.
Robot learning from demonstration by constructing skill trees
Konidaris, G., Kuindersma, S., Grupen, R., and Barto, A · 2012
Earlier work this paper cites.
Learning and generalization of complex tasks from unstructured demonstrations
Niekum, S., Osentoski, S., Konidaris, G., and Barto, A. G · 2012
Earlier work this paper cites.

Similar

The principle of maximum causal entropy for estimating interacting processes
Ziebart, B. D., Bagnell, J. A., and Dey, A. K · 2012
Cited alongside, same era.
Dynamical movement primitives: learning attractor models for motor behaviors
Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., and Schaal, S · 2013
Cited alongside, same era.
Auto-encoding variational bayes, 2013
Original
Kingma, D. P. and Welling, M · 2013
Cited alongside, same era.
Learning to select and generalize striking movements in robot table tennis
Mülling, K., Kober, J., Kroemer, O., and Peters, J · 2013
Cited alongside, same era.
Towards robot skill learning: From simple skills to table tennis
Peters, J., Kober, J., Mülling, K., Krämer, O., and Neumann, G · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Cited alongside, same era.
Learning modular policies for robotics
Neumann, G., Daniel, C., Paraschos, A., Kupcsik, A., and Peters, J · 2014
Cited alongside, same era.
Continuous control with deep reinforcement learning
Original
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D · 2015
Cited alongside, same era.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation
Kulkarni, T. D., Narasimhan, K. R., Saeedi, A., and Tenenbaum, J. B · 2016
Cited alongside, same era.
Tsc-dl: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with deep learning
Murali, A., Garg, A., Krishnan, S., Pokorny, F. T., Abbeel, P., Darrell, T., and Goldberg, K · 2016
Cited alongside, same era.
Modular multitask reinforcement learning with policy sketches
Andreas, J., Klein, D., and Levine, S · 2017
Cited alongside, same era.
The option-critic architecture
Bacon, P.-L., Harb, J., and Precup, D · 2017
Cited alongside, same era.

Then

Learning movement primitive libraries through probabilistic segmentation
Lioutikov, R., Neumann, G., Maeda, G., and Peters, J · 2017
Later among the works it cites.
Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings
Original
Co-Reyes, J. D., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., and Levine, S · 2018
Later among the works it cites.
Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning
Krishnan, S., Garg, A., Patil, S., Lea, C., Hager, G., Abbeel, P., and Goldberg, K · 2018
Later among the works it cites.
Roboturk: A crowdsourcing platform for robotic skill learning through imitation
Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., Savarese, S., and Fei-Fei, L · 2018
Later among the works it cites.
Multiple interactions made easy (mime): Large scale demonstrations data for imitation
Sharma, P., Mohan, L., Pinto, L., and Gupta, A · 2018
Later among the works it cites.
Taco: Learning task decomposition via temporal alignment for control
Original
Shiarlis, K., Wulfmeier, M., Salter, S., Whiteson, S., and Posner, I · 2018
Later among the works it cites.
An inference-based policy gradient method for learning options
Smith, M., Hoof, H., and Pineau, J · 2018
Later among the works it cites.
Neural task programming: Learning to generalize across hierarchical tasks
Xu, D., Nair, S., Zhu, Y., Gao, J., Garg, A., Fei-Fei, L., and Savarese, S · 2018
Later among the works it cites.
Temporal difference variational auto-encoder
Gregor, K., Papamakarios, G., Besse, F., Buesing, L., and Weber, T · 2019
Later among the works it cites.
Neural task graphs: Generalizing to unseen tasks from a single video demonstration
Huang, D.-A., Nair, S., Xu, D., Zhu, Y., Garg, A., Fei-Fei, L., Savarese, S., and Niebles, J. C · 2019
Later among the works it cites.
Variational temporal abstraction
Kim, T., Ahn, S., and Bengio, Y · 2019
Later among the works it cites.
Compile: Compositional imitation learning and execution
Kipf, T., Li, Y., Dai, H., Zambaldi, V., Sanchez-Gonzalez, A., Grefenstette, E., Kohli, P., and Battaglia, P · 2019
Later among the works it cites.
Directed-info GAIL: learning hierarchical policies from unsegmented demonstrations using directed information
Sharma, M., Sharma, A., Rhinehart, N., and Kitani, K. M · 2019
Later among the works it cites.
Learning attribute grammars for movement primitive sequencing
Lioutikov, R., Maeda, G., Veiga, F., Kersting, K., and Peters, J · 2020
Closest in time.
Discovering motor programs by recomposing demonstrations
Shankar, T., Tulsiani, S., Pinto, L., and Gupta, A · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…