2021

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Agarwal, Rishabh, Machado, Marlos C., Castro, Pablo Samuel et al.

Understand

Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments.

To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process.
This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly.
Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states.

Built on

Bisimulation through probabilistic testing
Kim G Larsen and Arne Skou · 1991
Earlier work this paper cites.
Extrapolation limitations of multilayer feedforward neural networks
Pamela J Haley and DONALD Soloway · 1992
Earlier work this paper cites.
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Martin L Puterman · 1994
Earlier work this paper cites.
Equivalence notions and model minimization in markov decision processes
Robert Givan, Thomas Dean, and Matthew Greig · 2003
Earlier work this paper cites.
Curl: Contrastive unsupervised representations for reinforcement learning
Original
Michael Laskin, Aravind Srinivas, and Pieter Abbeel · 2003
Earlier work this paper cites.
Metrics for finite Markov decision processes
Norm Ferns, Prakash Panangaden, and Doina Precup · 2004
Earlier work this paper cites.
Reinforcement learning with augmented data
Original
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas · 2004
Earlier work this paper cites.
Methods for computing state similarity in Markov decision processes
Norm Ferns, Pablo Samuel Castro, Doina Precup, and Prakash Panangaden · 2006
Earlier work this paper cites.
Dimensionality reduction by learning an invariant mapping
Raia Hadsell, Sumit Chopra, and Yann LeCun · 2006
Earlier work this paper cites.
Predictive information accelerates learning in rl
Original
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama · 2007
Earlier work this paper cites.
Optimal transport: old and new
Cédric Villani · 2008
Earlier work this paper cites.
Transfer learning for reinforcement learning domains: A survey
Matthew E. Taylor and Peter Stone · 2009
Earlier work this paper cites.
Using bisimulation for policy transfer in MDPs
Pablo Samuel Castro and Doina Precup · 2010
Earlier work this paper cites.
Bisimulation metrics for continuous markov decision processes
Norm Ferns, Prakash Panangaden, and Doina Precup · 2011
Earlier work this paper cites.
Bisimulation metrics are optimal value functions
Norman Ferns and Doina Precup · 2014
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Earlier work this paper cites.
Google vizier: A service for black-box optimization
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley · 2017
Earlier work this paper cites.
Implicit regularization in matrix factorization
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro · 2017
Earlier work this paper cites.
Robust and efficient transfer learning with hidden parameter Markov decision processes
Taylor W. Killian, George Dimitri Konidaris, and Finale Doshi-Velez · 2017
Earlier work this paper cites.
Zero-shot task generalization with multi-task deep reinforcement learning
Junhyuk Oh, Satinder P. Singh, Honglak Lee, and Pushmeet Kohli · 2017
Earlier work this paper cites.
The 2017 davis challenge on video object segmentation
Original
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool · 2017
Earlier work this paper cites.

Similar

Towards generalization and simplicity in continuous control
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham M. Kakade · 2017
Cited alongside, same era.
Domain randomization for transferring deep neural networks from simulation to the real world
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel · 2017
Cited alongside, same era.
Metareg: Towards domain generalization using meta-regularization
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa · 2018
Cited alongside, same era.
Generalization and regularization in DQN
Jesse Farebrother, Marlos C. Machado, and Michael Bowling · 2018
Cited alongside, same era.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Original
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine · 2018
Cited alongside, same era.
Illuminating generalization in deep reinforcement learning through procedural level generation
Original
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi · 2018
Cited alongside, same era.
Learning to generalize: Meta-learning for domain generalization
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales · 2018
Cited alongside, same era.
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
L. McInnes, J. Healy, and J. Melville · 2018
Cited alongside, same era.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Cited alongside, same era.
Assessing generalization in deep reinforcement learning
Original
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song · 2018
Cited alongside, same era.
Learning invariances for policy generalization
Remi Tachet des Combes, Philip Bachman, and Harm van Seijen · 2018
Cited alongside, same era.
Measuring and characterizing generalization in deep reinforcement learning
Sam Witty, Jun Ki Lee, Emma Tosch, Akanksha Atrey, Michael Littman, and David Jensen · 2018
Cited alongside, same era.

Then

Scalable methods for computing state similarity in deterministic Markov decision processes
Pablo Samuel Castro · 2020
Later among the works it cites.
A simple framework for contrastive learning of visual representations
Original
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton · 2020
Later among the works it cites.
Domain extrapolation via regret minimization
Original
Wengong Jin, Regina Barzilay, and Tommi Jaakkola · 2020
Later among the works it cites.
Image augmentation is all you need: Regularizing deep reinforcement learning from pixels
Original
Ilya Kostrikov, Denis Yarats, and Rob Fergus · 2020
Later among the works it cites.
Deep reinforcement and infomax learning
Bogdan Mazoure, Remi Tachet des Combes, Thang Long DOAN, Philip Bachman, and R Devon Hjelm · 2020
Later among the works it cites.
Learning to score behaviors for guided policy optimization
Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, and Michael Jordan · 2020
Later among the works it cites.
Generalized hidden parameter mdps transferable model-based rl in a handful of trials
Christian F. Perez, Felipe Petroski Such, and Theofanis Karaletsos · 2020
Later among the works it cites.
Automatic data augmentation for generalization in deep reinforcement learning
Original
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus · 2020
Later among the works it cites.
Data-efficient reinforcement learning with momentum predictive representations
Original
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman · 2020
Later among the works it cites.
Invariant policy optimization: Towards stronger generalization in reinforcement learning
Original
Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar · 2020
Later among the works it cites.
Decoupling representation learning from reinforcement learning
Original
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin · 2020
Later among the works it cites.
Neuroevolution of self-interpretable agents
Original
Yujin Tang, Duong Nguyen, and David Ha · 2020
Later among the works it cites.
dm_control: Software and tasks for continuous control
Original
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess · 2020
Later among the works it cites.
Plannable approximations to mdp homomorphisms: Equivariance under actions
Elise van der Pol, Thomas Kipf, Frans A Oliehoek, and Max Welling · 2020
Later among the works it cites.
How neural networks extrapolate: From feedforward to graph neural networks
Original
Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka · 2020
Later among the works it cites.
Rotation, translation, and cropping for zero-shot generalization
Original
Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius · 2020
Later among the works it cites.
Invariant causal prediction for block mdps
Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup · 2020
Later among the works it cites.
Efficient wasserstein natural gradients for reinforcement learning
Ted Moskovitz, Michael Arbel, Ferenc Huszar, and Arthur Gretton · 2021
Closest in time.
The distracting control suite – a challenging benchmark for reinforcement learning from pixels
Original
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski · 2021
Closest in time.
Learning invariant representations for reinforcement learning without reconstruction
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…