Fetching the paper…
Reading the bibliography…
Recent work has shown the promise of creating generalist, transformer-based, models for language, vision, and sequential decision-making problems.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.
Layer normalization, 2016
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton · 2016
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas · 2016
Earlier work this paper cites.
Pointer sentinel mixture models, 2016
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2016
Earlier work this paper cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017
Earlier work this paper cites.
Attention is all you need, 2017
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford and Karthik Narasimhan · 2018
Earlier work this paper cites.
D4rl: Datasets for deep data-driven reinforcement learning, 2020
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine · 2020
Earlier work this paper cites.
Towards continual reinforcement learning: A review and perspectives, 2020
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup · 2020
Earlier work this paper cites.
Decision transformer: Reinforcement learning via sequence modeling, 2021
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch · 2021
Earlier work this paper cites.
The role of permutation invariance in linear mode connectivity of neural networks, 2021
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur · 2021
Earlier work this paper cites.
Generalized decision transformer for offline hindsight information matching, 2021
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu · 2021
Earlier work this paper cites.
Pretrained transformers as universal computation engines, 2021
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch · 2021
Earlier work this paper cites.
Merging models with fisher-weighted averaging, 2021
Michael Matena and Colin Raffel · 2021
Cited alongside, same era.
Robust fine-tuning of zero-shot models, 2021
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt · 2021
Cited alongside, same era.
Git re-basin: Merging models modulo permutation symmetries, 2022
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa · 2022
Cited alongside, same era.
Pact: Perception-action causal transformer for autoregressive robotics pre-training, 2022
Rogerio Bonatti, Sai Vemprala, Shuang Ma, Felipe Frujeri, Shuhang Chen, and Ashish Kapoor · 2022
Cited alongside, same era.
Fusing finetuned models for better pretraining, 2022
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz · 2022
Cited alongside, same era.
Cold fusion: Collaborative descent for distributed multitask finetuning, 2022
Multi-game decision transformers, 2022
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch · 2022
Later among the works it cites.
Branch-train-merge: Embarrassingly parallel training of expert language models, 2022
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer · 2022
Later among the works it cites.
Re-basin via implicit sinkhorn differentiation, 2022
Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli · 2022
Later among the works it cites.
Formal algorithms for transformers
Mary Phuong and Marcus Hutter · 2022
Later among the works it cites.
A generalist agent
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen · 2022
Cited alongside, same era.
Editing models with task arithmetic, 2022
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi · 2022
Cited alongside, same era.
Patching open-vocabulary models by interpolating weights, 2022
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt · 2022
Cited alongside, same era.
Vima-manipulation, 2022
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan · 2022
Cited alongside, same era.
Dataless knowledge fusion by merging weights of language models, 2022
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng · 2022
Cited alongside, same era.
Repair: Renormalizing permuted activations for interpolation repair
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur · 2022
Cited alongside, same era.
Offline q-learning on diverse multi-task data both scales and generalizes, 2022
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine · 2022
Cited alongside, same era.
Can wikipedia help offline reinforcement learning?, 2022
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu · 2022
Later among the works it cites.
On the effect of pre-training for transformer in different modality on offline reinforcement learning
Shiro Takagi · 2022
Later among the works it cites.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt · 2022
Later among the works it cites.
On the feasibility of cross-task transfer with model-based reinforcement learning, 2022
Yifan Xu, Nicklas Hansen, Zirui Wang, Yung-Chieh Chan, Hao Su, and Zhuowen Tu · 2022
Later among the works it cites.
Adaptersoup: Weight averaging to improve generalization of pretrained language models, 2023
Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, and Jesse Dodge · 2023
Closest in time.
Learning universal policies via text-guided video generation, 2023
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel · 2023
Closest in time.
Investigating multi-task pretraining and generalization in reinforcement learning
Adrien Ali Taiga, Rishabh Agarwal, Jesse Farebrother, Aaron Courville, and Marc G Bellemare · 2023
Closest in time.