Fetching the paper…

Reading the bibliography…

2023

PolyTask: Learning Unified Policies through Behavior Distillation

Haldar, Siddhant, Pinto, Lerrel

Original
PDF
Post on X

Understand

Unified models capable of solving a wide variety of tasks have gained traction in vision and NLP due to their ability to share regularities and structures across tasks, which improves individual task performance and reduces computational footprint.

However, the impact of such models remains limited in embodied learning problems, which present unique challenges due to interactivity, sample inefficiency, and sequential task presentation.
In this work, we present PolyTask, a novel method for learning a single unified model that can solve various embodied tasks through a 'learn then distill' mechanism.
In the 'learn' step, PolyTask leverages a few demonstrations for each task to train task-specific policies.

PolyTask: Learning Unified Policies through Behavior Distillation · Around

Built on

N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in
Original
1908
Earlier work this paper cites.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in
Original
1910
Earlier work this paper cites.
J. S. Vitter, “Random sampling with a reservoir,”
1985
Earlier work this paper cites.
A. Robins, “Catastrophic forgetting in neural networks: the role of rehearsal mechanisms,” in
1993
Earlier work this paper cites.
R. Caruana, “Multitask learning,”
1997
Earlier work this paper cites.
S. Thrun, “Lifelong learning algorithms.”
1998
Earlier work this paper cites.
S. M. Barnett and S. J. Ceci, “When and where do we apply what we learn?: A taxonomy for far transfer.”
2002
Earlier work this paper cites.
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in
2010
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in
2012
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
Original
2014
Earlier work this paper cites.
A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,”
Original
2015
Earlier work this paper cites.
E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Actor-mimic: Deep multitask and transfer reinforcement learning,”
Original
2015
Earlier work this paper cites.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,”
Original
2016
Earlier work this paper cites.
Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu, “Distral: Robust multitask reinforcement learning,”
2017
Earlier work this paper cites.
G. I. Parisi, J. Tani, C. Weber, and S. Wermter, “Lifelong learning of human actions with deep neural network self-organization,”
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”
2017
Earlier work this paper cites.
J. D. Power and B. L. Schlaggar, “Neural plasticity across the lifespan,”
2017
Earlier work this paper cites.
Z. Li and D. Hoiem, “Learning without forgetting,”
2017
Earlier work this paper cites.
W. Ying, Y. Zhang, J. Huang, and Q. Yang, “Transfer learning via learning to transfer,” in
2018
Earlier work this paper cites.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq,
Original
2018
Earlier work this paper cites.
Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in
2018
Earlier work this paper cites.
J. Mendez, S. Shivkumar, and E. Eaton, “Lifelong inverse reinforcement learning,”
2018
Earlier work this paper cites.

Similar

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,”
2018
Cited alongside, same era.
L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon, “Unified language model pre-training for natural language understanding and generation,”
2019
Cited alongside, same era.
W. M. Czarnecki, R. Pascanu, S. Osindero, S. Jayakumar, G. Swirszcz, and M. Jaderberg, “Distilling policy distillation,” in
2019
Cited alongside, same era.
M. Ryu, Y. Chow, R. Anderson, C. Tjandraatmadja, and C. Boutilier, “Caql: Continuous action q-learning,”
Original
2019
Cited alongside, same era.
A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman, “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,”
Original

Then

S. Sodhani, L. Denoyer, P.-A. Kamienny, and O. Delalleau, “Mtenv - environment interface for mulit-task reinforcement learning,” Github, 2021. [Online]. Available:
2021
Later among the works it cites.
D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Mt-opt: Continuous multi-task robotic reinforcement learning at scale,”
Original
2021
Later among the works it cites.
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg,
Original
2022
Later among the works it cites.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu,
Original
2022
Later among the works it cites.
J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi, “Unified-io: A unified model for vision, language, and multi-modal tasks,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2019

Cited alongside, same era.

G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,”

2019

Cited alongside, same era.

H. Bao, L. Dong, F. Wei, W. Wang, N. Yang, X. Liu, Y. Wang, J. Gao, S. Piao, M. Zhou,

2020

Cited alongside, same era.

R. Jena, C. Liu, and K. Sycara, “Augmenting gail with bc for sample efficient imitation learning,”

2020

Cited alongside, same era.

T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn, “Gradient surgery for multi-task learning,”

2020

Cited alongside, same era.

N. Vithayathil Varghese and Q. H. Mahmoud, “A survey of multi-task deep reinforcement learning,”

2020

Cited alongside, same era.

R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman, “Efficient adaptation for end-to-end vision-based robotic manipulation,” in

2020

Cited alongside, same era.

C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in

2020

Cited alongside, same era.

2022

Later among the works it cites.

A. Kolesnikov, A. Susano Pinto, L. Beyer, X. Zhai, J. Harmsen, and N. Houlsby, “Uvim: A unified modeling approach for vision with learned guiding codes,”

2022

Later among the works it cites.

I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao,

Original

2022

Later among the works it cites.

L. Smith, I. Kostrikov, and S. Levine, “A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning,”

2022

Later among the works it cites.

H. Hoshino, K. Ota, A. Kanezaki, and R. Yokota, “Opirl: Sample efficient off-policy inverse reinforcement learning via distribution matching,” in

2022

Later among the works it cites.

C. Xu and J. McAuley, “A survey on model compression for natural language processing,”

2022

Later among the works it cites.

S. Cohen, B. Amos, M. P. Deisenroth, M. Henaff, E. Vinitsky, and D. Yarats, “Imitation learning from pixel observations for continuous control,” 2022. [Online]. Available:

2022

Later among the works it cites.

A. Xie and C. Finn, “Lifelong robotic reinforcement learning by retaining experiences,” in

2022

Later among the works it cites.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu,

2023

Closest in time.

M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in

2023

Closest in time.

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,”

2023

Closest in time.

S. Haldar, V. Mathur, D. Yarats, and L. Pinto, “Watch and match: Supercharging imitation with regularized optimal transport,” in

2023

Closest in time.

S. Haldar, J. Pari, A. Rai, and L. Pinto, “Teach a robot to fish: Versatile imitation from one minute of demonstrations,”

2023

Closest in time.

P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient online reinforcement learning with offline data,”

2023

Closest in time.

P. Wu, A. Majumdar, K. Stone, Y. Lin, I. Mordatch, P. Abbeel, and A. Rajeswaran, “Masked trajectory models for prediction, representation, and control,” in

2023

Closest in time.

A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P. Berges, P. Abbeel, J. Malik,

2023

Closest in time.

S. Auddy, J. Hollenstein, M. Saveriano, A. Rodríguez-Sánchez, and J. Piater, “Continual learning from demonstration of robotics skills,”

2023

Closest in time.