2020

Action and Perception as Divergence Minimization

Hafner, Danijar, Ortega, Pedro A., Ba, Jimmy et al.

Understand

To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions.

Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation.
However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans.
We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents.

Built on

Efficient Exploration via State Marginal Matching
Original
L Lee, B Eysenbach, E Parisotto, E Xing, S Levine, R Salakhutdinov · 1906
Earlier work this paper cites.
Stochastic Latent Actor-Critic: Deep Reinforcement Learning With a Latent Variable Model
Original
AX Lee, A Nagabandi, P Abbeel, S Levine · 1907
Earlier work this paper cites.
Dream to Control: Learning Behaviors by Latent Imagination
Original
D Hafner, T Lillicrap, J Ba, M Norouzi · 1912
Earlier work this paper cites.
Some Tests of Significance, Treated by the Theory of Probability
H Jeffreys · 1935
Earlier work this paper cites.
What Is Life? The Physical Aspect of the Living Cell and Mind
E Schrödinger · 1944
Earlier work this paper cites.
An Essentially Complete Class of Admissible Decision Functions
A Wald · 1947
Earlier work this paper cites.
A Mathematical Theory of Communication
CE Shannon · 1948
Earlier work this paper cites.
Cybernetics or Control and Communication in the Animal and the Machine
N Wiener · 1948
Earlier work this paper cites.
On Information and Sufficiency
S Kullback RA Leibler · 1951
Earlier work this paper cites.
A Method for the Construction of Minimum-Redundancy Codes
DA Huffman · 1952
Earlier work this paper cites.
Theory of Games and Economic Behavior
O Morgenstern J Von Neumann · 1953
Earlier work this paper cites.
On a Measure of the Information Provided by an Experiment
DV Lindley et al · 1956
Earlier work this paper cites.
Information Theory and Statistical Mechanics
ET Jaynes · 1957
Earlier work this paper cites.
The Principles of Quantum Mechanics
PAM Dirac · 1958
Earlier work this paper cites.
A New Approach to Linear Filtering and Prediction Problems
RE Kalman · 1960
Earlier work this paper cites.
Markov’s Conditional Processes
R Stratonovich · 1960
Earlier work this paper cites.
An Introduction to Cybernetics
WR Ashby · 1961
Earlier work this paper cites.
Risk Aversion in the Small and in the Large
JW Pratt · 1964
Earlier work this paper cites.
Complex Analysis, 1966
R Rudin · 1966
Earlier work this paper cites.
A Theory of Adaptive Pattern Classifiers
S Amari · 1967
Earlier work this paper cites.
Risk-Sensitive Markov Decision Processes
RA Howard JE Matheson · 1972
Earlier work this paper cites.
Perceptions as Hypotheses
RL Gregory · 1980
Earlier work this paper cites.
A Complete Class Theorem for Statistical Problems With Finite Sample Spaces
LD Brown · 1981
Earlier work this paper cites.
The Science of Structure: Synergetics
H Haken · 1981
Earlier work this paper cites.
Large Automatic Learning, Rule Extraction, and Generalization
J Denker, D Schwartz, B Wittner, S Solla, R Howard, L Jackel, J Hopfield · 1987
Earlier work this paper cites.
A Mean Field Theory Learning Algorithm for Neural Networks
C Peterson · 1987
Earlier work this paper cites.
Stochastic Multisensory Data Fusion for Mobile Robot Location and Environment Modelling. 5th Int
P Moutarlier R Chatila · 1989
Earlier work this paper cites.
Curious Model-Building Control Systems
J Schmidhuber · 1991
Earlier work this paper cites.
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
RS Sutton · 1991
Earlier work this paper cites.
Function Optimization Using Connectionist Reinforcement Learning Algorithms
RJ Williams J Peng · 1991
Earlier work this paper cites.
Ockham’s Razor and Bayesian Analysis
WH Jefferys JO Berger · 1992
Earlier work this paper cites.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
RJ Williams · 1992
Earlier work this paper cites.
Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights
GE Hinton D Van Camp · 1993
Earlier work this paper cites.
The Helmholtz Machine
P Dayan, GE Hinton, RM Neal, RS Zemel · 1995
Earlier work this paper cites.
Learning From Incomplete Data, 1995
Z Ghahramani MI Jordan · 1995
Earlier work this paper cites.
Bayes Factors
RE Kass AE Raftery · 1995
Earlier work this paper cites.
Causal Diagrams for Empirical Research
J Pearl · 1995
Earlier work this paper cites.
An Introduction to Variational Methods for Graphical Models
MI Jordan, Z Ghahramani, TS Jaakkola, LK Saul · 1999
Earlier work this paper cites.
Between Mdps and Semi-Mdps: A Framework for Temporal Abstraction in Reinforcement Learning
RS Sutton, D Precup, S Singh · 1999
Earlier work this paper cites.
An Analysis of Stochastic Game Theory for Multiagent Reinforcement Learning
M Bowling M Veloso · 2000
Earlier work this paper cites.
Policy Gradient Methods for Reinforcement Learning With Function Approximation
RS Sutton, DA McAllester, SP Singh, Y Mansour · 2000
Earlier work this paper cites.
The IM Algorithm: a Variational Approach to Information Maximization
D Barber FV Agakov · 2003
Earlier work this paper cites.
Information Projections Revisited
I Csiszár F Matus · 2003
Earlier work this paper cites.
Information Theory, Inference and Learning Algorithms
DJ MacKay · 2003
Earlier work this paper cites.
Empowerment: A Universal Agent-Centric Measure of Control
AS Klyubin, D Polani, CL Nehaniv · 2005
Earlier work this paper cites.
Pattern Recognition and Machine Learning
CM Bishop · 2006
Earlier work this paper cites.
A Fast Learning Algorithm for Deep Belief Nets
GE Hinton, S Osindero, YW Teh · 2006
Earlier work this paper cites.
A Tutorial on Energy-Based Learning
Y LeCun, S Chopra, R Hadsell, M Ranzato, F Huang · 2006
Earlier work this paper cites.
Intrinsic Motivation Systems for Autonomous Mental Development
PY Oudeyer, F Kaplan, VV Hafner · 2007
Earlier work this paper cites.
General Duality Between Optimal Control and Estimation
E Todorov · 2008
Earlier work this paper cites.

Similar

Optimal Control as a Graphical Model Inference Problem
HJ Kappen, V Gómez, M Opper · 2009
Cited alongside, same era.
The Free-Energy Principle: a Unified Brain Theory?
K Friston · 2010
Cited alongside, same era.
Noise-Contrastive Estimation: a New Estimation Principle for Unnormalized Statistical Models
M Gutmann A Hyvärinen · 2010
Cited alongside, same era.
Deep Auto-Encoder Neural Networks in Reinforcement Learning
S Lange M Riedmiller · 2010
Cited alongside, same era.
A Minimum Relative Entropy Principle for Learning and Acting
PA Ortega DA Braun · 2010
Cited alongside, same era.
Approximate Inference and Stochastic Optimal Control
Original
K Rawlik, M Toussaint, S Vijayakumar · 2010
Cited alongside, same era.
Information, Utility and Bounded Rationality
DA Ortega PA Braun · 2011
Cited alongside, same era.
Planning to Be Surprised: Optimal Bayesian Exploration in Dynamic Environments
Y Sun, F Gomez, J Schmidhuber · 2011
Cited alongside, same era.
What Is Value–Accumulated Reward or Evidence?
K Friston, R Adams, R Montague · 2012
Cited alongside, same era.
Machine Learning: a Probabilistic Perspective
KP Murphy · 2012
Cited alongside, same era.
Dynamical Decomposition of Markov Processes Without Detailed Balance
P Ao, C Tian-Qi, S Jiang-Hong · 2013
Cited alongside, same era.
Life as We Know It
K Friston · 2013
Cited alongside, same era.

Then

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
Original
F Ebert, C Finn, S Dasari, A Xie, A Lee, S Levine · 2018
Later among the works it cites.
Neural Scene Representation and Rendering
SA Eslami, DJ Rezende, F Besse, F Viola, AS Morcos, M Garnelo, A Ruderman, AA Rusu, I Danihelka, K Gregor, et al · 2018
Later among the works it cites.
Diversity Is All You Need: Learning Skills Without a Reward Function
Original
B Eysenbach, A Gupta, J Ibarz, S Levine · 2018
Later among the works it cites.
Neural Predictive Belief Representations
Original
ZD Guo, MG Azar, B Piot, BA Pires, T Pohlen, R Munos · 2018
Later among the works it cites.
World Models
Original
D Ha J Schmidhuber · 2018
Later among the works it cites.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning With a Stochastic Actor
Original
T Haarnoja, A Zhou, P Abbeel, S Levine · 2018
Later among the works it cites.
Learning Latent Dynamics for Planning From Pixels
Original
D Hafner, T Lillicrap, I Fischer, R Villegas, D Ha, H Lee, J Davidson · 2018
Later among the works it cites.
Learning an Embedding Space for Transferable Robot Skills
K Hausman, JT Springenberg, Z Wang, N Heess, M Riedmiller · 2018
Later among the works it cites.
Averaging Weights Leads to Wider Optima and Better Generalization
Original
P Izmailov, D Podoprikhin, T Garipov, D Vetrov, AG Wilson · 2018
Later among the works it cites.
Semi-Amortized Variational Autoencoders
Original
Y Kim, S Wiseman, AC Miller, D Sontag, AM Rush · 2018
Later among the works it cites.
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
Original
S Levine · 2018
Later among the works it cites.
Approximate Bayesian Inference in Spatial Environments
Original
A Mirchev, B Kayalibay, M Soelch, P van der Smagt, J Bayer · 2018
Later among the works it cites.
Representation Learning With Contrastive Predictive Coding
Original
Avd Oord, Y Li, O Vinyals · 2018
Later among the works it cites.
Episodic Curiosity Through Reachability
Original
N Savinov, A Raichuk, R Marinier, D Vincent, M Pollefeys, T Lillicrap, S Gelly · 2018
Later among the works it cites.
Model-Based Active Exploration
Original
P Shyam, W Jaśkowski, F Gomez · 2018
Later among the works it cites.
Reinforcement Learning: An Introduction
RS Sutton AG Barto · 2018
Later among the works it cites.
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
Original
Y Wen, P Vicol, J Ba, D Tran, R Grosse · 2018
Later among the works it cites.
Noisy Natural Gradient as Variational Inference
G Zhang, S Sun, D Duvenaud, R Grosse · 2018
Later among the works it cites.
World Discovery Models
Original
MG Azar, B Piot, BA Pires, JB Grill, F Altché, R Munos · 2019
Later among the works it cites.
Smirl: Surprise Minimizing Rl in Dynamic Environments
Original
G Berseth, D Geng, C Devin, C Finn, D Jayaraman, S Levine · 2019
Later among the works it cites.
Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
Original
O Chang, Y Yao, D Williams-King, H Lipson · 2019
Later among the works it cites.
A Free Energy Principle for a Particular Physics
Original
K Friston · 2019
Later among the works it cites.
Information Asymmetry in Kl-Regularized Rl
Original
A Galashov, SM Jayakumar, L Hasenclever, D Tirumala, J Schwarz, G Desjardins, WM Czarnecki, YW Teh, R Pascanu, N Heess · 2019
Later among the works it cites.
A Divergence Minimization Perspective on Imitation Learning Methods
Original
SKS Ghasemipour, R Zemel, S Gu · 2019
Later among the works it cites.
Shaping Belief States With Generative Environment Models for Rl
Original
K Gregor, DJ Rezende, F Besse, Y Wu, H Merzic, Avd Oord · 2019
Later among the works it cites.
A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
F Leibfried, S Pascual-Diaz, J Grau-Moya · 2019
Later among the works it cites.
Generalised Free Energy and Active Inference
T Parr KJ Friston · 2019
Later among the works it cites.
Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
Original
VH Pong, M Dalal, S Lin, A Nair, S Bahl, S Levine · 2019
Later among the works it cites.
On Variational Bounds of Mutual Information
Original
B Poole, S Ozair, Avd Oord, AA Alemi, G Tucker · 2019
Later among the works it cites.
Dynamics-Aware Unsupervised Discovery of Skills
Original
A Sharma, S Gu, S Levine, V Kumar, K Hausman · 2019
Later among the works it cites.
Functional Variational Bayesian Neural Networks
Original
S Sun, G Zhang, J Shi, R Grosse · 2019
Later among the works it cites.
Exploiting Hierarchy for Learning and Transfer in Kl-Regularized Rl
Original
D Tirumala, H Noh, A Galashov, L Hasenclever, A Ahuja, G Wayne, R Pascanu, YW Teh, N Heess · 2019
Later among the works it cites.
Bayesian Layers: A Module for Neural Network Uncertainty
D Tran, M Dusenberry, M van der Wilk, D Hafner · 2019
Later among the works it cites.
On Mutual Information Maximization for Representation Learning
Original
M Tschannen, J Djolonga, PK Rubenstein, S Gelly, M Lucic · 2019
Later among the works it cites.
Improving Sample Efficiency in Model-Free Reinforcement Learning From Images
Original
D Yarats, A Zhang, I Kostrikov, B Amos, J Pineau, R Fergus · 2019
Later among the works it cites.
Solar: Deep Structured Representations for Model-Based Reinforcement Learning
M Zhang, S Vikram, L Smith, P Abbeel, M Johnson, S Levine · 2019
Later among the works it cites.
Efficient Online Estimation of Empowerment for Reinforcement Learning
Original
R Zhao, P Abbeel, S Tiomkin · 2019
Later among the works it cites.
A Simple Framework for Contrastive Learning of Visual Representations
Original
T Chen, S Kornblith, M Norouzi, G Hinton · 2020
Closest in time.
Active Inference on Discrete State-Spaces: a Synthesis
Original
L Da Costa, T Parr, N Sajid, S Veselic, V Neacsu, K Friston · 2020
Closest in time.
Efficient and Scalable Bayesian Neural Nets With Rank-1 Factors
Original
MW Dusenberry, G Jerfel, Y Wen, Ya Ma, J Snoek, K Heller, B Lakshminarayanan, D Tran · 2020
Closest in time.
The Conditional Entropy Bottleneck
Original
I Fischer · 2020
Closest in time.
Sophisticated Inference
Original
K Friston, L Da Costa, D Hafner, C Hesp, T Parr · 2020
Closest in time.
Improving Predictions of Bayesian Neural Networks via Local Linearization
Original
A Immer, M Korzepa, M Bauer · 2020
Closest in time.
Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
Original
A Kirsch, C Lyle, Y Gal · 2020
Closest in time.
Making Sense of Reinforcement Learning and Probabilistic Inference
Original
B O’Donoghue, I Osband, C Ionescu · 2020
Closest in time.
Planet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
Original
M Okada, N Kosaka, T Taniguchi · 2020
Closest in time.
Planning to Explore via Self-Supervised World Models
Original
R Sekar, O Rybkin, K Daniilidis, P Abbeel, D Hafner, D Pathak · 2020
Closest in time.
Learning Robot Skills With Temporal Variational Inference
Original
T Shankar A Gupta · 2020
Closest in time.
Exploration Entropy for Reinforcement Learning
B Xin, H Yu, Y Qin, Q Tang, Z Zhu · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…