2021

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

Nair, Suraj, Mitchell, Eric, Chen, Kevin et al.

Understand

We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction.

In order to accomplish this, humans need easy and effective ways of specifying tasks to the robot.
Goal images are one popular form of task specification, as they are already grounded in the robot's observation space.
However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks.

Built on

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Original
M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox · 1912
Earlier work this paper cites.
Understanding natural language
T. Winograd · 1972
Earlier work this paper cites.
Learning to achieve goals
L. P. Kaelbling · 1993
Earlier work this paper cites.
Guiding a reinforcement learner with natural language advice: Initial results in robocup soccer
G. Kuhlmann, P. Stone, R. J. Mooney, and Shavlik · 2004
Earlier work this paper cites.
Walk the talk: Connecting language, knowledge, and action in route instructions
M. MacMahon, B. Stankiewicz, and B. Kuipers · 2006
Earlier work this paper cites.
Toward understanding natural language directions
T. Kollar, S. Tellex, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Grounding verbs of motion in natural language commands to robots
T. Kollar, S. Tellex, D. Roy, and N. Roy · 2010
Earlier work this paper cites.
Learning to interpret natural language navigation instructions from observations
D. L. Chen and R. J. Mooney · 2011
Earlier work this paper cites.
Understanding natural language commands for robotic navigation and mobile manipulation
S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy · 2011
Earlier work this paper cites.
Weakly supervised learning of semantic parsers for mapping instructions to actions
Y. Artzi and L. Zettlemoyer · 2013
Earlier work this paper cites.
The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning
R. Y. Rubinstein and D. P. Kroese · 2013
Earlier work this paper cites.
Alignment-based compositional semantics for instruction following
J. Andreas and D. Klein · 2015
Earlier work this paper cites.
Learning language games through interaction
Original
S. I. Wang, P. Liang, and C. D. Manning · 2016
Earlier work this paper cites.
Listen, attend, and walk: Neural mapping of navigational instructions to action sequences
H. Mei, M. Bansal, and M. R. Walter · 2016
Earlier work this paper cites.
Deep visual foresight for planning robot motion
C. Finn and S. Levine · 2017
Earlier work this paper cites.
Modular multitask reinforcement learning with policy sketches
Original
J. Andreas, D. Klein, and S. Levine · 2017
Earlier work this paper cites.
Programmable agents
Original
M. Denil, S. G. Colmenarejo, S. Cabi, D. Saxton, and N. D. Freitas · 2017
Earlier work this paper cites.
Accurately and efficiently interpreting human-robot instructions of varying granularities
D. Arumugam, S. Karamcheti, N. Gopalan, L. Wong, and S. Tellex · 2017
Earlier work this paper cites.
Zero-shot task generalization with multi-task deep reinforcement learning
Original
J. Oh, S. Singh, H. Lee, and P. Kohli · 2017
Earlier work this paper cites.
Grounded language learning in a simulated 3d world
Original
K. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. Czarnecki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom · 2017
Earlier work this paper cites.
Matterport3d: Learning from RGB-D data in indoor environments
A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang · 2017
Earlier work this paper cites.
Mapping instructions and visual observations to actions with reinforcement learning
D. K. Misra, J. Langford, and Y. Artzi · 2017
Earlier work this paper cites.
Beating atari with natural language guided reinforcement learning
Original
R. Kaplan, C. Sauer, and A. Sosa · 2017
Earlier work this paper cites.
One-shot visual imitation learning via meta-learning
C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine · 2017
Earlier work this paper cites.
Hindsight experience replay
Original
M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. H. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba · 2017
Earlier work this paper cites.
Stochastic variational video prediction
Original
M. Babaeizadeh, C. Finn, D. Erhan, R. H. Campbell, and S. Levine · 2017
Earlier work this paper cites.

Similar

Visual reinforcement learning with imagined goals
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine · 2018
Cited alongside, same era.
Curious: Intrinsically motivated multi-task, multi-goal reinforcement learning
Original
C. Colas, P. Fournier, O. Sigaud, and P.-Y. Oudeyer · 2018
Cited alongside, same era.
Representation learning for grounded spatial reasoning
M. Jänner, K. Narasimhan, and R. Barzilay · 2018
Cited alongside, same era.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel · 2018
Cited alongside, same era.
Gated-attention architectures for task-oriented language grounding
D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov · 2018
Cited alongside, same era.
One-shot imitation from observing humans via domain-adaptive meta-learning
T. Yu, C. Finn, S. Dasari, A. Xie, T. Zhang, P. Abbeel, and S. Levine · 2018
Cited alongside, same era.
Visual foresight: Model-based deep reinforcement learning for vision-based robotic control
Original
F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine · 2018
Cited alongside, same era.
Variational inverse control with events: A general framework for data-driven reward definition
J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine · 2018
Cited alongside, same era.
The unreasonable effectiveness of deep features as a perceptual metric
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang · 2018
Cited alongside, same era.
From language to goals: Inverse reinforcement learning for vision-based instruction following
Original
J. Fu, A. Balan, S. Levine, and S. Guadarrama · 2019
Cited alongside, same era.
A survey of reinforcement learning informed by natural language
J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel · 2019
Cited alongside, same era.
Grounding natural language instructions to semantic goal representations for abstraction and generalization
D. Arumugam, S. Karamcheti, N. Gopalan, E. C. Williams, M. Rhee, L. L. S. Wong, and S. Tellex · 2019
Cited alongside, same era.

Then

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter
Original
V. Sanh, L. Debut, J. Chaumond, and T. Wolf · 2019
Later among the works it cites.
Broadly-exploring, local-policy trees for long-horizon task planning
Original
B. Ichter, P. Sermanet, and C. Lynch · 2020
Later among the works it cites.
Grounding language in play
Original
C. Lynch and P. Sermanet · 2020
Later among the works it cites.
Language-conditioned imitation learning for robot manipulation tasks
Original
S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. B. Amor · 2020
Later among the works it cites.
Human instruction-following with deep reinforcement learning via transfer-learning from text
Original
F. Hill, S. Mokra, N. Wong, and T. Harley · 2020
Later among the works it cites.
Program guided agent
S.-H. Sun, T.-L. Wu, and J. J. Lim · 2020
Later among the works it cites.
Topological planning with transformers for vision-and-language navigation
Original
K. Chen, J. Chen, J. Chuang, M. V’azquez, and S. Savarese · 2020
Later among the works it cites.
Beyond the nav-graph: Vision-and-language navigation in continuous environments
J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee · 2020
Later among the works it cites.
Following instructions by imagining and reaching visual goals
Original
J. D. Kanu, E. Dessalene, X. Lin, C. Fermuller, and Y. Aloimonos · 2020
Later among the works it cites.
Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards
Original
P. Goyal, S. Niekum, and R. Mooney · 2020
Later among the works it cites.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine · 2020
Later among the works it cites.
Meld: Meta-reinforcement learning from images via latent state models
Original
T. Zhao, A. Nagabandi, K. Rakelly, C. Finn, and S. Levine · 2020
Later among the works it cites.
Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation
Original
S. Nair and C. Finn · 2020
Later among the works it cites.
Program synthesis guided reinforcement learning
Original
Y. Yang, J. Inala, O. Bastani, Y. Pu, A. Solar-Lezama, and M. Rinard · 2021
Closest in time.
Hierarchical program-triggered reinforcement learning agents for automated driving
Original
B. Gangopadhyay, H. Soora, and P. Dasgupta · 2021
Closest in time.
Reinforcement learning of implicit and explicit control flow in instructions
E. Brooks, J. Rajendran, R. L. Lewis, and S. Singh · 2021
Closest in time.
Grounding Language to Autonomously-Acquired Skills via Goal Generation
A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud · 2021
Closest in time.
Ella: Exploration through learned language abstraction
Original
S. P. Mirchandani, S. Karamcheti, and D. Sadigh · 2021
Closest in time.
Batch exploration with examples for scalable robotic reinforcement learning
A. S. Chen, H. Nam, S. Nair, and C. Finn · 2021
Closest in time.
Mt-opt: Continuous multi-task robotic reinforcement learning at scale
Original
D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman · 2021
Closest in time.
Learning generalizable robotic reward functions from ”in-the-wild” human videos
Original
A. S. Chen, S. Nair, and C. Finn · 2021
Closest in time.
Disco rl: Distribution-conditioned reinforcement learning for general-purpose policies
Original
S. Nasiriany, V. H. Pong, A. Nair, A. Khazatsky, G. Berseth, and S. Levine · 2021
Closest in time.
Actionable models: Unsupervised offline reinforcement learning of robotic skills
Original
Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. C. Julian, C. Finn, and S. Levine · 2021
Closest in time.
Model-based visual planning with self-supervised functional distances
S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine · 2021
Closest in time.
Greedy hierarchical variational autoencoders for large-scale video prediction
Original
B.-H. Wu, S. Nair, R. Martín-Martín, L. Fei-Fei, and C. Finn · 2021
Closest in time.
Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks
B. Wu, S. Nair, L. Fei-Fei, and C. Finn · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…