2023

Policy Representation via Diffusion Probability Model for Reinforcement Learning

Yang, Long, Huang, Zhixiong, Lei, Fenghao et al.

Understand

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration.

The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL.
In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL.
Concretely, we character diffusion policy as a stochastic process, which is a new approach to representing a policy.

Built on

Die mittlere energie rotierender elektrischer dipole im strahlungsfeld
Adriaan Daniël Fokker · 1914
Earlier work this paper cites.
Über einen satz der statistischen dynamik und seine erweiterung in der quantentheorie
VM Planck · 1917
Earlier work this paper cites.
Über die analytischen methoden in der wahrscheinlichkeitsrechnung
Andrei Kolmogoroff · 1931
Earlier work this paper cites.
Reverse-time diffusion equation models
Brian DO Anderson · 1982
Earlier work this paper cites.
Asymptotic evaluation of certain markov process expectations for large time. iv
Monroe D Donsker and SR Srinivasa Varadhan · 1983
Earlier work this paper cites.
Time reversal of diffusions
Ulrich G Haussmann and Etienne Pardoux · 1986
Earlier work this paper cites.
Alvinn: An autonomous land vehicle in a neural network
Dean A Pomerleau · 1988
Earlier work this paper cites.
Learning to predict by the methods of temporal differences
Richard S Sutton · 1988
Earlier work this paper cites.
Learning from delayed rewards
Christopher John Cornish Hellaby Watkins · 1989
Earlier work this paper cites.
Introduction to stochastic integration , volume 2
Kai Lai Chung and Ruth J Williams · 1990
Earlier work this paper cites.
On-line Q-learning using connectionist systems , volume 37
Gavin A Rummery and Mahesan Niranjan · 1994
Earlier work this paper cites.
Fokker-planck equation
Hannes Risken and Hannes Risken · 1996
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 1998
Earlier work this paper cites.
Policy gradient methods for reinforcement learning with function approximation
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour · 1999
Earlier work this paper cites.
Reinforcement learning with factored states and actions
Brian Sallans and Geoffrey E Hinton · 2004
Earlier work this paper cites.
Estimation of non-normalized statistical models by score matching
Aapo Hyvärinen · 2005
Earlier work this paper cites.
A tutorial on energy-based learning
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc’Aurelio Ranzato, and Fu Jie Huang · 2006
Earlier work this paper cites.
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
A survey of robot learning from demonstration
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning · 2009
Earlier work this paper cites.
Double q-learning
Hado Hasselt · 2010
Earlier work this paper cites.
Relative entropy policy search
Jan Peters, Katharina Mulling, and Yasemin Altun · 2010
Earlier work this paper cites.
A connection between score matching and denoising autoencoders
Pascal Vincent · 2011
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Emanuel Todorov, Tom Erez, and Yuval Tassa · 2012
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Analysis and geometry of Markov diffusion operators , volume 103
Dominique Bakry, Ivan Gentil, Michel Ledoux, et al · 2014
Earlier work this paper cites.
Deterministic policy gradient algorithms
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller · 2014
Earlier work this paper cites.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Danilo Rezende and Shakir Mohamed · 2015
Earlier work this paper cites.
Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli · 2015
Earlier work this paper cites.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Earlier work this paper cites.
Continuous control with deep reinforcement learning
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra · 2016
Earlier work this paper cites.

Similar

Reinforcement learning with deep energy-based policies
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
Neural discrete representation learning
Aaron Van Den Oord, Oriol Vinyals, et al · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Cited alongside, same era.
Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al · 2018
Cited alongside, same era.
Learning robust rewards with adverserial inverse reinforcement learning
Justin Fu, Katie Luo, and Sergey Levine · 2018
Cited alongside, same era.
Addressing function approximation error in actor-critic methods
Scott Fujimoto, Herke Hoof, and David Meger · 2018
Cited alongside, same era.
Composable deep reinforcement learning for robotic manipulation
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine · 2018
Cited alongside, same era.
An algorithmic perspective on imitation learning
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al · 2018
Cited alongside, same era.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 2018
Cited alongside, same era.
Convexity of mutual information along the ornstein-uhlenbeck flow
Andre Wibisono and Varun Jog · 2018
Cited alongside, same era.
Sampling can be faster than optimization
Yi-An Ma, Yuansi Chen, Chi Jin, Nicolas Flammarion, and Michael I Jordan · 2019
Cited alongside, same era.
Mish: A self regularized non-monotonic activation function
Original
Diganta Misra · 2019
Cited alongside, same era.

Then

Planning with diffusion for flexible behavior synthesis
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine · 2022
Later among the works it cites.
Dall-e-bot: Introducing web-scale diffusion models to robotics
Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns · 2022
Later among the works it cites.
Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation
Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon · 2022
Later among the works it cites.
Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects
Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris Paxton · 2022
Later among the works it cites.
Trajectory balance: Improved credit assignment in gflownets
Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio · 2022
Later among the works it cites.
What matters in language conditioned robotic imitation learning over unstructured data
Oier Mees, Lukas Hermann, and Wolfram Burgard · 2022
Later among the works it cites.
A generalist agent
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas · 2022
Later among the works it cites.
Behavior transformers: Cloning k k modes with one stone
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto · 2022
Later among the works it cites.
Hierarchical policies for cluttered-scene grasping with latent plans
Lirui Wang, Xiangyun Meng, Yu Xiang, and Dieter Fox · 2022
Later among the works it cites.
Convergence in kl divergence of the inexact langevin algorithm with application to score-based generative models
Original
Andre Wibisono and Kaylee Yingxi Yang · 2022
Later among the works it cites.
Diffusion models: A comprehensive survey of methods and applications
Original
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang · 2022
Later among the works it cites.
Fast sampling of diffusion models with exponential integrator
Chen Yongxin Zhang, Qinsheng · 2022
Later among the works it cites.
Is conditional generative modeling all you need for decision-making?
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal · 2023
Closest in time.
Edgi: Equivariant diffusion for planning with embodied agents
Johann Brehmer, Joey Bose, Pim De Haan, and Taco Cohen · 2023
Closest in time.
Diffusion policy: Visuomotor policy learning via action diffusion
Original
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song · 2023
Closest in time.
From play to policy: Conditional behavior generation from uncurated robot data
Zichen Jeff Cui, Yibin Wang, Nur Muhammad, Lerrel Pinto, et al · 2023
Closest in time.
Arc-actor residual critic for adversarial imitation learning
Ankur Deka, Changliu Liu, and Katia P Sycara · 2023
Closest in time.
Preference transformer: Modeling human preferences using transformers for rl
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee · 2023
Closest in time.
Contrastive decision transformers
Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay · 2023
Closest in time.
Convergence of score-based generative modeling for general data distributions
Holden Lee, Jianfeng Lu, and Yixin Tan · 2023
Closest in time.
Adaptdiffuser: Diffusion models as adaptive self-evolving planners
Original
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo · 2023
Closest in time.
Reorientdiff: Diffusion model based reorientation for object manipulation
Original
Utkarsh A Mishra and Yongxin Chen · 2023
Closest in time.
Stochastic generative flow networks
Original
Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio · 2023
Closest in time.
Imitating human behaviour with diffusion models
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al · 2023
Closest in time.
Model-based adversarial imitation learning as online fine-tuning
Rafael Rafailov, Victor Kolev, Kyle Beltran Hatch, John D Martin, Mariano Phielipp, Jiajun Wu, and Chelsea Finn · 2023
Closest in time.
Goal-conditioned imitation learning using score-based diffusion policies
Original
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov · 2023
Closest in time.
Latent plans for task-agnostic offline reinforcement learning
Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard · 2023
Closest in time.
Adversarial imitation learning with preferences
Aleksandar Taranovic, Andras Gabor Kupcsik, Niklas Freymuth, and Gerhard Neumann · 2023
Closest in time.
Se(3)-diffusionfields: Learning cost functions for joint grasp and motion optimization through diffusion
Julen Urain, Niklas Funk, Georgia Chalvatzaki, and Jan Peters · 2023
Closest in time.
Diffusion policies as an expressive policy class for offline reinforcement learning
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou · 2023
Closest in time.
Foundation models for decision making: Problems, methods, and opportunities
Original
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…