2016

Learning from the Hindsight Plan -- Episodic MPC Improvement

Tamar, Aviv, Thomas, Garrett, Zhang, Tianhao et al.

Understand

Model predictive control (MPC) is a popular control method that has proved effective for robotics, among other fields.

MPC performs re-planning at every time step.
Re-planning is done with a limited horizon per computational and real-time constraints and often also for robustness to potential model errors.
However, the limited horizon leads to suboptimal performance.

Built on

J. Rust, “Maximum likelihood estimation of discrete control processes,” SIAM Journal on Control and Optimization , vol. 26, no. 5, pp. 1006–1024, 1988
1988
Earlier work this paper cites.
D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large scale optimization,” Mathematical programming , 1989
1989
Earlier work this paper cites.
H. Chen and F. ALLGoWER, “A quasi-infinite horizon nonlinear model predictive control scheme with guaranteed stability,” Automatica , vol. 34, no. 10, pp. 1205–1217, 1998
1998
Earlier work this paper cites.
K. L. Moore and J.-X. Xu, Iterative learning control . Taylor & Francis, 2000
2000
Earlier work this paper cites.
R. W. Longman, “Iterative learning control and repetitive control for engineering practice,” International journal of control , vol. 73, no. 10, pp. 930–954, 2000
2000
Earlier work this paper cites.
E. K. Chong, R. L. Givan, and H. S. Chang, “A framework for simulation-based network control via hindsight optimization,” in CDC , 2000
2000
Earlier work this paper cites.
P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in ICML . ACM, 2004, p. 1
2004
Earlier work this paper cites.
D. A. Bristow, M. Tharayil, and A. G. Alleyne, “A survey of iterative learning control,” IEEE Control Systems , 2006
2006
Earlier work this paper cites.
P. Abbeel, A. Coates, M. Quigley, and A. Y. Ng, “An application of reinforcement learning to aerobatic helicopter flight,” NIPS , 2007
2007
Earlier work this paper cites.
B. D. Anderson and J. B. Moore, Optimal control: linear quadratic methods . Courier Corporation, 2007
2007
Earlier work this paper cites.
Y. Wang, F. Gao, and F. J. Doyle, “Survey on iterative learning control, repetitive control, and run-to-run control,” Journal of Process Control , vol. 19, no. 10, pp. 1589–1600, 2009
2009
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks.” in Aistats , vol. 9, 2010, pp. 249–256
2010
Earlier work this paper cites.

Similar

M. Deisenroth and C. E. Rasmussen, “PILCO: A model-based and data-efficient approach to policy search,” in ICML , 2011, pp. 465–472
2011
Cited alongside, same era.
A. Aswani, P. Bouffard, and C. Tomlin, “Extensions of learning-based model predictive control for real-time application to a quadrotor helicopter,” in ACC , 2012, pp. 4661–4666
2012
Cited alongside, same era.
T. Erez, Y. Tassa, and E. Todorov, “Infinite-horizon model predictive control for periodic tasks with contacts,” Robotics: Science and systems , p. 73, 2012
2012
Cited alongside, same era.
M. Zucker and J. A. Bagnell, “Reinforcement planning: RL for optimal planners,” in ICRA . IEEE, 2012, pp. 1850–1855
2012
Cited alongside, same era.
Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient backprop,” in Neural networks: Tricks of the trade . Springer, 2012
2012
Cited alongside, same era.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems . IEEE, 2012, pp. 5026–5033
2012
Cited alongside, same era.
E. F. Camacho and C. B. Alba, Model predictive control . Springer Science & Business Media, 2013
2013
Cited alongside, same era.
T. Erez, K. Lowrey, Y. Tassa, V. Kumar, S. Kolev, and E. Todorov, “An integrated system for real-time model predictive control of humanoid robots,” in Humanoids , 2013
2013
Cited alongside, same era.
G. Chowdhary, M. Mühlegg, J. P. How, and F. Holzapfel, “Concurrent learning adaptive model predictive control,” in Advances in Aerospace Guidance, Navigation and Control . Springer, 2013, pp. 29–47
2013
Cited alongside, same era.
J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” International Journal of Robotics Research , 2013
2013
Cited alongside, same era.
M. Zhong, M. Johnson, Y. Tassa, T. Erez, and E. Todorov, “Value function approximation and model predictive control,” in 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) . IEEE, 2013, pp. 100–107
2013
Cited alongside, same era.
K. J. Åström and B. Wittenmark, Adaptive control . Courier Corporation, 2013
2013
Cited alongside, same era.

Then

S. Levine and P. Abbeel, “Learning neural network policies with guided policy search under unknown dynamics,” in NIPS , 2014
2014
Later among the works it cites.
I. Lenz, R. Knepper, and A. Saxena, “Deepmpc: Learning deep latent features for model predictive control,” in Robotics Science and Systems , 2015
2015
Later among the works it cites.
N. Jiang, A. Kulesza, S. Singh, and R. Lewis, “The dependence of effective planning horizon on model accuracy,” in AAMAS . International Foundation for Autonomous Agents and Multiagent Systems, 2015, pp. 1181–1189
2015
Later among the works it cites.
S. Levine, N. Wagener, and P. Abbeel, “Learning contact-rich manipulation skills with guided policy search,” in ICRA , 2015
2015
Later among the works it cites.
J. Schulman, N. Heess, T. Weber, and P. Abbeel, “Gradient estimation using stochastic computation graphs,” in NIPS , 2015, pp. 3528–3536
2015
Later among the works it cites.
J. Fu, S. Levine, and P. Abbeel, “One-shot learning of manipulation skills with online dynamics adaptation and neural network priors,” IROS , 2016
2016
Closest in time.
U. Rosolia and F. Borrelli, “Learning model predictive control for iterative tasks,” arXiv preprint arXiv:1609.01387 , 2016
Original
2016
Closest in time.
A. Tamar, Y. Wu, G. Thomas, S. Levine, and P. Abbeel, “Value iteration networks,” CoRR , vol. abs/1602.02867, 2016
Original
2016
Closest in time.
C. Finn, M. Zhang, J. Fu, X. Tan, Z. McCarthy, E. Scharff, and S. Levine, “Guided policy search code implementation,” 2016. [Online]. Available: http://rll.berkeley.edu/gps
2016
Closest in time.
M. Zhang, Z. McCarthy, C. Finn, S. Levine, and P. Abbeel, “Learning deep neural network policies with continuous memory states,” in ICRA . IEEE, 2016, pp. 520–527
2016
Closest in time.
Y. Pan, X. Yan, E. Theodorou, and B. Boots, “Adaptive probabilistic trajectory optimization via efficient approximate inference,” arXiv preprint arXiv:1608.06235 , 2016
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…