2018

State Representation Learning for Control: An Overview

Lesort, Timothée, Díaz-Rodríguez, Natalia, Goudou, Jean-François et al.

Understand

Representation learning algorithms are designed to learn abstract features that characterize data.

State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent.
The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios.
In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning.

Built on

Introduction to reinforcement learning
Sutton, R. S. (1998) · 1998
Earlier work this paper cites.
Algorithmic applications of low-distortion geometric embeddings
Indyk, P. (2001) · 2001
Earlier work this paper cites.
A survey of dimension reduction techniques
Fodor, I. K. (2002) · 2002
Earlier work this paper cites.
Slow feature analysis: Unsupervised learning of invariances
Wiskott, L. and Sejnowski, T. J. (2002) · 2002
Earlier work this paper cites.
Learning a similarity metric discriminatively, with application to face verification
Chopra, S., Hadsell, R., and LeCun, Y. (2005) · 2005
Earlier work this paper cites.
Empowerment: A universal agent-centric measure of control
Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005) · 2005
Earlier work this paper cites.
Learning to control an octopus arm with gaussian process temporal difference methods
Engel, Y., Szabo, P., and Volkinshtein, D. (2006) · 2006
Earlier work this paper cites.
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., and Teh, Y. W. (2006) · 2006
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Oudeyer, P.-Y., Kaplan, F., and Hafner, V. (2007) · 2007
Earlier work this paper cites.
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. (2008) · 2008
Earlier work this paper cites.
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., and Manzagol, P.-A. (2010) · 2010
Earlier work this paper cites.
Pilco: A model-based and data-efficient approach to policy search
Deisenroth, M. P. and Rasmussen, C. E. (2011) · 2011
Earlier work this paper cites.
Incremental slow feature analysis: Adaptive and episodic learning from high-dimensional input streams
Original
Kompella, V. R., Luciw, M. D., and Schmidhuber, J. (2011) · 2011
Earlier work this paper cites.
Unsupervised feature learning and deep learning: A review and new perspectives
Original
Bengio, Y., Courville, A. C., and Vincent, P. (2012) · 2012
Earlier work this paper cites.
The Mario AI benchmark and competitions
Karakovskiy, Sergey, T. J. (2012) · 2012
Earlier work this paper cites.
Learn to swing up and balance a real pole based on raw visual input data
Mattner, J., Lange, S., and Riedmiller, M. A. (2012) · 2012
Earlier work this paper cites.
A folded neural network autoencoder for dimensionality reduction
Wang, J., He, H., and Prokhorov, D. V. (2012) · 2012
Earlier work this paper cites.
A new embedding quality assessment method for manifold learning
Zhang, P., Ren, Y., and Zhang, B. (2012) · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013) · 2013
Earlier work this paper cites.
Auto-Encoding Variational Bayes
Kingma, D. P. and Welling, M. (2013) · 2013
Earlier work this paper cites.
Crossing the reality gap: a short introduction to the transferability approach
Original
Mouret, J., Koos, S., and Doncieux, S. (2013) · 2013
Earlier work this paper cites.
Robot Skill Learning: From Reinforcement Learning to Evolution Strategies
Stulp, F. and Sigaud, O. (2013) · 2013
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014) · 2014
Earlier work this paper cites.
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
Jimenez Rezende, D., Mohamed, S., and Wierstra, D. (2014) · 2014
Earlier work this paper cites.
Data-efficient learning of feedback policies from image pixels using deep dynamical models
Assael, J.-A. M., Wahlström, N., Schön, T. B., and Deisenroth, M. P. (2015) · 2015
Earlier work this paper cites.

Similar

Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations
Böhmer, W., Springenberg, J. T., Boedecker, J., Riedmiller, M., and Obermayer, K. (2015) · 2015
Cited alongside, same era.
Using PCA to Efficiently Represent State Spaces
Curran, W., Brys, T., Taylor, M., and Smart, W. (2015) · 2015
Cited alongside, same era.
Deep spatial autoencoders for visuomotor learning
Original
Finn, C., Tan, X. Y., Duan, Y., Darrell, T., Levine, S., and Abbeel, P. (2015) · 2015
Cited alongside, same era.
Learning to linearize under uncertainty
Original
Goroshin, R., Mathieu, M., and LeCun, Y. (2015) · 2015
Cited alongside, same era.
Learning state representations with robotic priors
Jonschkowski, R. and Brock, O. (2015) · 2015
Cited alongside, same era.
Deep Kalman Filters
Krishnan, R. G., Shalit, U., and Sontag, D. (2015) · 2015
Cited alongside, same era.
Continuous control with deep reinforcement learning
Original
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015) · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · 2015
Cited alongside, same era.
From Pixels to Torques: Policy Learning with Deep Dynamical Models
Wahlström, N., Schön, T. B., and Deisenroth, M. P. (2015) · 2015
Cited alongside, same era.
Embed to control: A locally linear latent dynamics model for control from raw images
Watter, M., Springenberg, J., Boedecker, J., and Riedmiller, M. (2015) · 2015
Cited alongside, same era.
Learning to poke by poking: Experiential learning of intuitive physics
Original
Agrawal, P., Nair, A., Abbeel, P., Malik, J., and Levine, S. (2016) · 2016
Cited alongside, same era.
Deepmind lab
Original
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., and Petersen, S. (2016) · 2016
Cited alongside, same era.

Then

Progressive neural networks
Original
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. (2016) · 2016
Later among the works it cites.
Stable reinforcement learning with autoencoders for tactile and visual data
van Hoof, H., Chen, N., Karl, M., van der Smagt, P., and Peters, J. (2016) · 2016
Later among the works it cites.
Auto-encoder based dimensionality reduction
Wang, Y., Yao, H., and Zhao, S. (2016) · 2016
Later among the works it cites.
A Separation Principle for Control in the Age of Deep Learning
Achille, A. and Soatto, S. (2017) · 2017
Later among the works it cites.
Autoencoder-augmented Neuroevolution for Visual Doom Playing
Alvernaz, S. and Togelius, J. (2017) · 2017
Later among the works it cites.
Interactive perception: Leveraging action in perception and perception in action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., and Sukhatme, G. S. (2017) · 2017
Later among the works it cites.
Using simulation and domain adaptation to improve efficiency of deep robotic grasping
Original
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2017) · 2017
Later among the works it cites.
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
Conti, E., Madhavan, V., Petroski Such, F., Lehman, J., Stanley, K. O., and Clune, J. (2017) · 2017
Later among the works it cites.
Learning state representations for robotic control
Duan, W. (2017) · 2017
Later among the works it cites.
Deep reinforcement learning that matters
Original
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2017) · 2017
Later among the works it cites.
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
Higgins, I., Pal, A., Rusu, A. A., Matthey, L., Burgess, C. P., Pritzel, A., Botvinick, M., Blundell, C., and Lerchner, A. (2017) · 2017
Later among the works it cites.
PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
Original
Jonschkowski, R., Hafner, R., Scholz, J., and Riedmiller, M. A. (2017) · 2017
Later among the works it cites.
Unsupervised state representation learning with robotic priors: a robustness benchmark
Original
Lesort, T., Seurin, M., Li, X., Díaz-Rodríguez, N., and Filliat, D. (2017) · 2017
Later among the works it cites.
Value Prediction Network
Oh, J., Singh, S., and Lee, H. (2017) · 2017
Later among the works it cites.
Goal-driven dimensionality reduction for reinforcement learning
Parisi, S., Ramstedt, S., and J., P. (2017) · 2017
Later among the works it cites.
Curiosity-driven exploration by self-supervised prediction
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017) · 2017
Later among the works it cites.
Time-contrastive networks: Self-supervised learning from multi-view observation
Original
Sermanet, P., Lynch, C., Hsu, J., and Levine, S. (2017) · 2017
Later among the works it cites.
Loss is its own reward: Self-supervision for reinforcement learning
Original
Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T. (2017) · 2017
Later among the works it cites.
Independently controllable factors
Original
Thomas, V., Pondard, J., Bengio, E., Sarfati, M., Beaudoin, P., Meurs, M., Pineau, J., Precup, D., and Bengio, Y. (2017) · 2017
Later among the works it cites.
Deep multimodal representation learning from temporal data
Original
Yang, X., Ramesh, P., Chitta, R., Madhvanath, S., Bernal, E. A., and Luo, J. (2017) · 2017
Later among the works it cites.
World Models
Ha, D. and Schmidhuber, J. (2018) · 2018
Closest in time.
Curiosity-driven reinforcement learning with homeostatic regulation
Magrans de Abril, I. and Kanai, R. (2018) · 2018
Closest in time.
Unsupervised learning of goal spaces for intrinsically motivated goal exploration
Péré, A., Forestier, S., Oudeyer, P.-Y., and Sigaud, O. (2018) · 2018
Closest in time.
DeepMind Control Suite
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. (2018) · 2018
Closest in time.
Decoupling dynamics and reward for transfer learning
Zhang, A., Satija, H., and Pineau, J. (2018) · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…