2022

Grounding Hindsight Instructions in Multi-Goal Reinforcement Learning for Robotics

Röder, Frank, Eppe, Manfred, Wermter, Stefan

Understand

This paper focuses on robotic reinforcement learning with sparse rewards for natural language goal representations.

An open problem is the sample-inefficiency that stems from the compositionality of natural language, and from the grounding of language in sensory data and actions.
We address these issues with three contributions.
We first present a mechanism for hindsight instruction replay utilizing expert feedback.

Built on

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Conference on Neural Information Processing Systems , ser. NIPS’14, vol. 2. Cambridge, MA, USA: MIT Press, 2014, pp. 3104–3112
2014
Earlier work this paper cites.
K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” in Conference on Empirical Methods in Natural Language Processing , A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha, Qatar: Association for Computational Linguistics, 2014, pp. 1724–1734
2014
Earlier work this paper cites.
T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal Value Function Approximators,” in International Conference on Machine Learning , F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 2015, pp. 1312–1320
2015
Earlier work this paper cites.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” arXiv:1606.01540 [cs] , 2016
Original
2016
Earlier work this paper cites.
E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016
2016
Earlier work this paper cites.
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight Experience Replay,” in Advances in Neural Information Processing Systems , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., vol. 30. Long Beach, CA, USA: Curran Associates, Inc., 2017, pp. 5048–5058
2017
Earlier work this paper cites.
M. Eppe, M. Kerzel, S. Griffiths, H. G. Ng, and S. Wermter, “Combining deep learning for visuomotor coordination with object identification to realize a high-level interface for robot object-picking,” in International Conference on Humanoid Robotics . Birmingham, UK: IEEE, 2017, pp. 612–617
2017
Earlier work this paper cites.
D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov, “Gated-Attention Architectures for Task-Oriented Language Grounding,” in Conference on Artificial Intelligence , S. A. McIlraith and K. Q. Weinberger, Eds., vol. 8. New Orleans, Louisiana, USA: AAAI Press, 2018, pp. 2819–2826
2018
Earlier work this paper cites.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” in International Conference on Machine Learning , J. G. Dy and A. Krause, Eds., vol. 80. Stockholm, Sweden: PMLR, 2018, pp. 1861–1870
2018
Earlier work this paper cites.

Similar

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction , 2nd ed., ser. A Bradford Book. MIT press, 2018
2018
Cited alongside, same era.
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual Reinforcement Learning with Imagined Goals,” in Advances in Neural Information Processing Systems , S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Montréal, Canada: Curran Associates, Inc., 2018, pp. 9209–9220
2018
Cited alongside, same era.
J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel, “A Survey of Reinforcement Learning Informed by Natural Language,” in International Joint Conference on Artificial Intelligence . Macao, China: International Joint Conferences on Artificial Intelligence Organization, 2019, pp. 6309–6317
2019
Cited alongside, same era.
Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn, “Language as an Abstraction for Hierarchical Deep Reinforcement Learning,” in Advances in Neural Information Processing Systems , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., vol. 32. Vancouver, Canada: Curran Associates, Inc., 2019, pp. 9419–9431
2019
Cited alongside, same era.
M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio, “BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning,” in International Conference on Learning Representations . New Orleans, Louisiana, USA: OpenReview.net, 2019
2019
Cited alongside, same era.
Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto, and J. Turian, “Experience Grounds Language,” in Conference on Empirical Methods in Natural Language Processing , B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, 2020, pp. 8718–8735
2020
Cited alongside, same era.
S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots That Use Language,” Annual Review of Control, Robotics, and Autonomous Systems , vol. 3, no. 1, pp. 25–55, 2020
2020
Cited alongside, same era.
G. Cideron, M. Seurin, F. Strub, and O. Pietquin, “HIGhER: Improving instruction following with Hindsight Generation for Experience Replay,” in Symposium Series on Computational Intelligence . Canberra, ACT, Australia: IEEE, 2020, pp. 225–232
2020
Cited alongside, same era.
C. Colas, T. Karch, N. Lair, J.-M. Dussoux, C. Moulin-Frier, P. F. Dominey, and P.-Y. Oudeyer, “Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.-F. Balcan, and H.-T. Lin, Eds., vol. 33. Vancouver, Canada: Curran Associates, Inc., 2020, pp. 3761–3774
2020
Cited alongside, same era.
M. Kerzel, E. Strahl, S. Magg, N. Navarro-Guerrero, S. Heinrich, and S. Wermter, “NICO — Neuro-inspired companion: A developmental humanoid robot platform for multimodal interaction,” in International Symposium on Robot and Human Interactive Communication . Lisbon, Portugal: IEEE, 28, pp. 113–120
Cited in the paper.

Then

C. Lynch and P. Sermanet, “Language Conditioned Imitation Learning Over Unstructured Data,” in Robotics: Science and Systems , D. A. Shell, M. Toussaint, and M. A. Hsieh, Eds., Virtual, 2021
2021
Later among the works it cites.
A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud, “Grounding Language to Autonomously-Acquired Skills via Goal Generation,” in International Conference on Learning Representations . Virtual (formerly Vienna, Austria): OpenReview.net, 2021
2021
Later among the works it cites.
M. Shridhar, L. Manuelli, and D. Fox, “CLIPort: What and Where Pathways for Robotic Manipulation,” in Conference on Robot Learning , ser. Proceedings of Machine Learning Research, A. Faust, Ed. London, UK: PMLR, 2021, pp. 894–906
2021
Later among the works it cites.
F. Röder, O. Özdemir, P. D. H. Nguyen, S. Wermter, and M. Eppe, “The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review,” Frontiers in Psychology , vol. 12, p. 3374, 2021
2021
Later among the works it cites.
F. Hill, O. Tieleman, T. von Glehn, N. Wong, H. Merzic, and S. Clark, “Grounded Language Learning Fast and Slow,” in International Conference on Learning Representations . Virtual (formerly Vienna, Austria): OpenReview.net, 2021
2021
Later among the works it cites.
K. X. Nguyen, D. Misra, R. Schapire, M. Dudik, and P. Shafto, “Interactive Learning from Activity Description,” in International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. Virtual: PMLR, 2021, pp. 8096–8108
2021
Later among the works it cites.
Q. Gallouédec, N. Cazin, E. Dellandréa, and L. Chen, “Panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,” in Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS , Virtual, 2021
2021
Later among the works it cites.
C. Bai, L. Wang, Y. Wang, Z. Wang, R. Zhao, C. Bai, and P. Liu, “Addressing Hindsight Bias in Multigoal Reinforcement Learning,” IEEE Transactions on Cybernetics , pp. 1–14, 2021
2021
Later among the works it cites.
M. Eppe, C. Gumbsch, M. Kerzel, P. D. H. Nguyen, M. V. Butz, and S. Wermter, “Intelligent problem-solving as integrated hierarchical reinforcement learning,” Nature Machine Intelligence , vol. 4, no. 1, pp. 11–20, 2022
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…