2017

A Multi-Armed Bandit Approach for Online Expert Selection in Markov Decision Processes

Mazumdar, Eric, Dong, Roy, Royo, Vicenç Rúbies et al.

Understand

We formulate a multi-armed bandit (MAB) approach to choosing expert policies online in Markov decision processes (MDPs).

Given a set of expert policies trained on a state and action space, the goal is to maximize the cumulative reward of our agent.
The hope is to quickly find the best expert in our set.
The MAB formulation allows us to quantify the performance of an algorithm in terms of the regret incurred from not choosing the best expert from the beginning.

Regret analysis of stochastic and nonstochastic multi-armed bandit problems
Sébastien Bubeck and Nicolò Cesa-Bianchi · 1935
Earlier work this paper cites.
Some aspects of the sequential design of experiments
Herbert Robbins · 1952
Earlier work this paper cites.
Probability inequalities for sums of bounded random variables
Wassily Hoeffding · 1963
Earlier work this paper cites.
Weighted sums of certain dependent random variables
Kazuoki Azuma · 1967
Earlier work this paper cites.
A two-armed bandit theory of market pricing
Michael Rothschild · 1974
Earlier work this paper cites.
Asymptotically efficient adaptive allocation rules
Tze Leung Lai and Herbert Robbins · 1985
Earlier work this paper cites.
Optimal two-stage designs for phase ii clinical trials
Richard Simon · 1989
Earlier work this paper cites.
Finite-time analysis of the multiarmed bandit problem
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer · 2002
Earlier work this paper cites.

Foundations of Modern Probability
Olav Kallenberg · 2002
Cited alongside, same era.
Online sparse bandit for card games
David Lupien St-Pierre, Quentin Louveaux, and Olivier Teytaud · 2011
Cited alongside, same era.
Regret bounds for restless markov bandits
Original
Ronald Ortner, Daniil Ryabko, Peter Auer, and Rémi Munos · 2012
Cited alongside, same era.
Online learning of rested and restless bandits
C. Tekin and M. Liu · 2012
Cited alongside, same era.
Multi-armed recommendation bandits for selecting state machine policies for robotic systems
Pyry Matikainen, P. Michael Furlong, Rahul Sukthankar, and Martial Hebert · 2013
Cited alongside, same era.
Playing Atari with deep reinforcement learning
Original
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller · 2013
Cited alongside, same era.
Global optimality in tensor factorization, deep learning, and beyond
Benjamin D. Haeffele and Rene Vidal · 2015
Cited alongside, same era.
Trust region policy optimization
Original
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel
Cited in the paper.
High-dimensional continuous control using generalized advantage estimation
Original
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel
Cited in the paper.

Multi-armed bandit models for 2D grasp planning with uncertainty
Michael Laskey, Jeff Mahler, Zoe McCarthy, Florian T Pokorny, Sachin Patil, Jur Van Den Berg, Danica Kragic, Pieter Abbeel, and Ken Goldberg · 2015
Later among the works it cites.
End-to-end training of deep visuomotor policies
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel · 2016
Later among the works it cites.
Asynchronous Methods for Deep Reinforcement Learning
Original
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Deep exploration via bootstrapped DQN
Original
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy · 2016
Later among the works it cites.
Metacontrol for adaptive imagination-based optimization
Original
Jessica B. Hamrick, Andrew J. Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peter W. Battaglia · 2017
Closest in time.
On the expressive power of deep neural networks
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein · 2017
Closest in time.
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…