Fetching the paper…

Reading the bibliography…

2025

LUMOS: Language-Conditioned Imitation Learning with World Models

Nematollahi, Iman, DeMoss, Branton, Chandra, Akshay L et al.

Original
PDF
Post on X

Understand

We introduce LUMOS, a language-conditioned multi-task imitation learning framework for robotics.

LUMOS learns skills by practicing them over many long-horizon rollouts in the latent space of a learned world model and transfers these skills zero-shot to a real robot.
By learning on-policy in the latent space of the learned world model, our algorithm mitigates policy-induced distribution shift which most offline imitation learning methods suffer from.
LUMOS learns from unstructured play data with fewer than 1% hindsight language annotations but is steerable with language commands at test time.

LUMOS: Language-Conditioned Imitation Learning with World Models · Around

Built on

R. S. Sutton, “
1988
Earlier work this paper cites.
D. A. Pomerleau, “
1989
Earlier work this paper cites.
S. Harnad, “
1990
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, “
2005
Earlier work this paper cites.
S. Ross and J. A. Bagnell, “
2010
Earlier work this paper cites.
Y. Bengio, N. Léonard, and A. Courville, “
Original
2013
Earlier work this paper cites.
D. P. Kingma and M. Welling, “
Original
2013
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski,
2015
Earlier work this paper cites.
M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “
2017
Earlier work this paper cites.
T. Salimans, A. Karpathy, X. Chen, and D. P. Kingma, “
Original
2017
Earlier work this paper cites.
D. Ha and J. Schmidhuber, “
Original
2018
Earlier work this paper cites.
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “
Original
2018
Earlier work this paper cites.

Similar

J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “
Original
2019
Cited alongside, same era.
C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “
Original
2019
Cited alongside, same era.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “
Original
2019
Cited alongside, same era.
N. Reimers and I. Gurevych, “
Original
2019
Cited alongside, same era.
J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel,
2020
Cited alongside, same era.

Then

B. DeMoss, P. Duckworth, N. Hawes, and I. Posner, “
Original
2023
Later among the works it cites.
H. Zhou, Z. Bing, X. Yao, X. Su, C. Yang, K. Huang, and A. Knoll, “
Original
2023
Later among the works it cites.
O. Mees, J. Borja-Diaz, and W. Burgard, “
Original
2023
Later among the works it cites.
C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “
Original
2023
Later among the works it cites.
M. Reuss, M. Li, X. Jia, and R. Lioutikov, “
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “

2020

Cited alongside, same era.

C. Lynch and P. Sermanet, “

2021

Cited alongside, same era.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,

2021

Cited alongside, same era.

D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “

2021

Cited alongside, same era.

O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard, “

2022

Cited alongside, same era.

O. Mees, L. Hermann, and W. Burgard, “

2022

Cited alongside, same era.

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “

Original

2023

Cited alongside, same era.

Z. Xian, N. Gkanatsios, T. Gervet, T.-W. Ke, and K. Fragkiadaki, “

2023

Later among the works it cites.

K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine, “

2023

Later among the works it cites.

H. Ha, P. Florence, and S. Song, “

2023

Later among the works it cites.

P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg, “

Original

2023

Later among the works it cites.

T.-W. Ke, N. Gkanatsios, and K. Fragkiadaki, “

2024

Later among the works it cites.

M. Reuss, Ö. E. Yağmurlu, F. Wenzel, and R. Lioutikov, “

2024

Later among the works it cites.

E. Zhang, Y. Lu, S. Huang, W. Y. Wang, and A. Zhang, “

2024

Later among the works it cites.