Fetching the paper…

Reading the bibliography…

2023

Contrastive Language, Action, and State Pre-training for Robot Learning

Rana, Krishan, Melnik, Andrew, Sünderhauf, Niko

Original
PDF
Post on X

Understand

In this paper, we introduce a method for unifying language, action, and state information in a shared embedding space to facilitate a range of downstream tasks in robot learning.

Our method, Contrastive Language, Action, and State Pre-training (CLASP), extends the CLIP formulation by incorporating distributional learning, capturing the inherent complexities and one-to-many relationships in behaviour-text alignment.
By employing distributional outputs for both text and behaviour encoders, our model effectively associates diverse textual commands with a single behaviour and vice-versa.
We demonstrate the utility of our method for the following downstream tasks: zero-shot text-behaviour retrieval, captioning unseen robot behaviours, and learning a behaviour prior for language-conditioned reinforcement learning.

Contrastive Language, Action, and State Pre-training for Robot Learning · Around

Built on

D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” CoRR , vol. abs/1312.6114, 2013
Original
2013
Earlier work this paper cites.
K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, D. Hassabis, and P. Blunsom, “Grounded language learning in a simulated 3d world,” ArXiv , vol. abs/1706.06551, 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017
2017
Earlier work this paper cites.
L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” International Conference on Learning Representations (ICLR) , 2017
Original
2017
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Earlier work this paper cites.
A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine, “Parrot: Data-driven behavioral priors for reinforcement learning,” 2020
2020
Earlier work this paper cites.
K. Pertsch, Y. Lee, and J. J. Lim, “Accelerating reinforcement learning with learned skill priors,” in Conference on Robot Learning (CoRL) , 2020
2020
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning , 2021
2021
Earlier work this paper cites.
R. Mokady, “Clipcap: Clip prefix for image captioning,” ArXiv , vol. abs/2111.09734, 2021
Original
2021
Earlier work this paper cites.
M. Tang, Z. Wang, Z. Liu, F. Rao, D. Li, and X. Li, “Clip4caption: Clip for video caption,” Proceedings of the 29th ACM International Conference on Multimedia , 2021
2021
Earlier work this paper cites.

Similar

C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,” Robotics: Science and Systems , 2021
2021
Cited alongside, same era.
D. I. A. T. J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, F. Fischer, P. Georgiev, A. Goldin, T. Harley, F. Hill, P. C. Humphreys, A. Hung, J. Landon, T. P. Lillicrap, H. Merzic, A. Muldal, A. Santoro, G. Scully, T. von Glehn, G. Wayne, N. Wong, C. Yan, and R. Zhu, “Creating multimodal interactive agents with imitation and self-supervised learning,” ArXiv , vol. abs/2112.03763, 2021
Original
2021
Cited alongside, same era.
S. Nair, E. Mitchell, K. Chen, B. Ichter, S. Savarese, and C. Finn, “Learning language-conditioned robot behavior from offline data and crowd-sourced annotation,” in Conference on Robot Learning , 2021
2021
Cited alongside, same era.
A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Audioclip: Extending clip to image, text and audio,” 2021
2021
Cited alongside, same era.

Then

O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning over unstructured data,” IEEE Robotics and Automation Letters , vol. 7, pp. 11 205–11 212, 2022
2022
Later among the works it cites.
E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in Conference on Robot Learning , 2022
2022
Later among the works it cites.
L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar, “Minedojo: Building open-ended embodied agents with internet-scale knowledge,” in Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022
2022
Later among the works it cites.
T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson, “Robotic skill acquisition via instruction augmentation with vision-language models,” ArXiv , vol. abs/2211.11736, 2022
Original
2022

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , vol. abs/2101.00190, 2021

2021

Cited alongside, same era.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” ArXiv , vol. abs/2204.06125, 2022

2022

Cited alongside, same era.

T. Lüddecke and A. Ecker, “Image segmentation using text and image prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 7086–7096

2022

Cited alongside, same era.

B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl, “Language-driven semantic segmentation,” in International Conference on Learning Representations , 2022

2022

Cited alongside, same era.

M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in Conference on Robot Learning , 2022

2022

Cited alongside, same era.

C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, “Interactive language: Talking to robots in real time,” 2022

2022

Cited alongside, same era.

Later among the works it cites.

K. Rana, M. Xu, B. Tidd, M. Milford, and N. Sunderhauf, “Residual skill policies: Learning an adaptable skill-based action space for reinforcement learning for robotics,” in Conference on Robot Learning , 2022

2022

Later among the works it cites.

L. X. Shi, J. J. Lim, and Y. Lee, “Skill-based model-based reinforcement learning,” in Conference on Robot Learning , 2022

2022

Later among the works it cites.

OpenAI, “Gpt-4 technical report,” ArXiv , vol. abs/2303.08774, 2023

2023

Closest in time.

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” ArXiv , vol. abs/2301.12597, 2023

2023

Closest in time.

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. H. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. R. Florence, “Palm-e: An embodied multimodal language model,” ArXiv , vol. abs/2303.03378, 2023

Original

2023

Closest in time.

T. Carta, C. Romac, T. Wolf, S. Lamprier, O. Sigaud, and P.-Y. Oudeyer, “Grounding large language models in interactive environments with online reinforcement learning,” 2023

2023

Closest in time.