2019

A Narration-based Reward Shaping Approach using Grounded Natural Language Commands

Waytowich, Nicholas, Barton, Sean L., Lawhern, Vernon et al.

Understand

While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longstanding problem of reward sparsity.

This is especially true for tasks such as training an agent to play StarCraft II, a real-time strategy game where reward is only given at the end of a game which is usually very long.
While this problem can be addressed through reward shaping, such approaches typically require a human expert with specialized knowledge.
Inspired by the vision of enabling reward shaping through the more-accessible paradigm of natural-language narration, we develop a technique that can provide the benefits of reward shaping using natural language commands.

Built on

Reinforcement learning in the multi-robot domain
Matarić, M. J · 1997
Earlier work this paper cites.
Robot shaping: an experiment in behavior engineering
Dorigo, M. and Colombetti, M · 1998
Earlier work this paper cites.
Visualizing data using t-sne
Maaten, L. v. d. and Hinton, G · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., and Browning, B · 2009
Earlier work this paper cites.
Interactively Shaping Agents via Human Reinforcement: The TAMER Framework
Knox, W. B. and Stone, P · 2009
Earlier work this paper cites.
Learning to interpret natural language navigation instructions from observations
Chen, D. and Mooney, R · 2011
Earlier work this paper cites.
Weakly supervised learning of semantic parsers for mapping instructions to actions
Artzi, Y. and Zettlemoyer, L · 2013
Earlier work this paper cites.
Learning to parse natural language commands to a robot control system
Matuszek, C., Herbst, E., Zettlemoyer, L., and Fox, D · 2013
Earlier work this paper cites.
Grounding english commands to reward functions
MacGlashan, J., Babes-Vroman, M., desJardins, M., Littman, M., Muresan, S., Squire, S., Tellex, S., Arumugam, D., and Yang, L · 2015
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. a., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Earlier work this paper cites.

Similar

Deep reinforcement learning with a natural language action space
He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Ostendorf, M · 2016
Cited alongside, same era.
Listen, attend, and walk: Neural mapping of navigational instructions to action sequences
Mei, H., Bansal, M., and Walter, M · 2016
Cited alongside, same era.
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al · 2016
Cited alongside, same era.
Accurately and efficiently interpreting human-robot instructions of varying granularities
Original
Arumugam, D., Karamcheti, S., Gopalan, N., Wong, L. L. S., and Tellex, S · 2017
Cited alongside, same era.
Beating Atari with Natural Lagnuage Guided Reinforcement Learning
Kaplan, R., Sauer, C., and Sosa, A · 2017
Cited alongside, same era.
Starcraft II: A new challenge for reinforcement learning
Original
Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., Quan, J., Gaffney, S., Petersen, S., Simonyan, K., Schaul, T., van Hasselt, H., Silver, D., Lillicrap, T. P., Calderone, K., Keet, P., Brunasso, A., Lawrence, D., Ekermo, A., Repp, J., and Tsing, R · 2017
Cited alongside, same era.
Following high-level navigation instructions on a simulated quadcopter with imitation learning
Blukis, V., Brukhim, N., Bennett, A., Knepper, R. A., and Artzi, Y · 2018
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.

Then

Mapping instructions to actions in 3D environments with visual goal prediction
Misra, D., Bennett, A., Blukis, V., Niklasson, E., Shatkhin, M., and Artzi, Y · 2018
Later among the works it cites.
On reinforcement learning for full-length game of starcraft
Original
Pang, Z., Liu, R., Meng, Z., Zhang, Y., Yu, Y., and Lu, T · 2018
Later among the works it cites.
Follownet: Robot navigation by following natural language directions with deep reinforcement learning
Original
Shah, P., Fiser, M., Faust, A., Kew, J. C., and Hakkani-Tür, D · 2018
Later among the works it cites.
Robobarista: Object part based transfer of manipulation trajectories from crowd-sourcing in 3d pointclouds
Sung, J., Jin, S. H., and Saxena, A · 2018
Later among the works it cites.
Reward learning from narrated demonstrations
Tung, H.-Y., Harley, A. W., Huang, L.-K., and Fragkiadaki, K · 2018
Later among the works it cites.
Deep tamer: Interactive agent shaping in high-dimensional state spaces
Warnell, G., Waytowich, N., Lawhern, V., and Stone, P · 2018
Later among the works it cites.
From language to goals: Inverse reinforcement learning for vision-based instruction following
Fu, J., Korattikara, A., Levine, S., and Guadarrama, S · 2019
Closest in time.
AlphaStar: Mastering the Real-Time Strategy Game StarCraft II, 2019
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki, W. M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., Ewalds, T., Horgan, D., Kroiss, M., Danihelka, I., Agapiou, J., Oh, J., Dalibard, V., Choi, D., Sifre, L., Sulsky, Y., Vezhnevets, S., Molloy, J., Cai, T., Budden, D., Paine, T., Gulcehre, C., Wang, Z., Pfaff, T., Pohlen, T., Wu, Y., Yogatama, D., Cohen, J., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Apps, C., Kavukcuoglu, K., Hassabis, D., and Silver, D · 2019
Closest in time.
Grounding natural language commands to starcraft ii game states for narration-guided reinforcement learning
Waytowich, N., Barton, S. L., Lawhern, V., Stump, E., and Warnell, G · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…