2019

A Survey on Practical Applications of Multi-Armed and Contextual Bandits

Bouneffouf, Djallel, Rish, Irina

Understand

In recent years, multi-armed bandit (MAB) framework has attracted a lot of attention in various applications, from recommender systems and information retrieval to healthcare and finance, due to its stellar performance combined with certain attractive properties, such as learning from less feedback.

The multi-armed bandit field is currently flourishing, as novel problem settings and algorithms motivated by various practical applications are being introduced, building on top of the classical bandit problem.
This article aims to provide a comprehensive review of top recent developments in multiple real-life applications of the multi-armed bandit.
Specifically, we introduce a taxonomy of common MAB-based applications and summarize state-of-art for each of those domains.

Built on

Asymptotically efficient adaptive allocation rules
T. L. Lai and Herbert Robbins · 1985
Earlier work this paper cites.
Finite-time analysis of the multiarmed bandit problem
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer · 2002
Earlier work this paper cites.
The epoch-greedy algorithm for multi-armed bandits with side information
John Langford and Tong Zhang · 2008
Earlier work this paper cites.
Algorithm selection as a bandit problem with unbounded losses
Matteo Gagliolo and Jürgen Schmidhuber · 2010
Earlier work this paper cites.
A contextual-bandit approach to personalized news article recommendation
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire · 2010
Earlier work this paper cites.
Analysis of thompson sampling for the multi-armed bandit problem
Shipra Agrawal and Navin Goyal · 2012
Earlier work this paper cites.
A contextual-bandit algorithm for mobile context-aware recommender system
Djallel Bouneffouf, Amel Bouzeghoub, and Alda Lopes Gançarski · 2012
Earlier work this paper cites.
Thompson sampling for contextual bandits with linear payoffs
Shipra Agrawal and Navin Goyal · 2013
Earlier work this paper cites.
Contextual bandits for context-based information retrieval
Djallel Bouneffouf, Amel Bouzeghoub, and Alda Lopes Gançarski · 2013
Earlier work this paper cites.
Building bridges: Viewing active learning from the multi-armed bandit lens
Original
Ravi Ganti and Alexander G Gray · 2013
Earlier work this paper cites.
A neural networks committee for the contextual bandit problem
Robin Allesiardo, Raphaël Féraud, and Djallel Bouneffouf · 2014
Earlier work this paper cites.
Contextual bandit for active learning: Active thompson sampling
Djallel Bouneffouf, Romain Laroche, Tanguy Urvoy, Raphael Féraud, and Robin Allesiardo · 2014
Earlier work this paper cites.
Freshness-aware thompson sampling
Djallel Bouneffouf · 2014
Earlier work this paper cites.

Similar

Online feature selection and its applications
Jialei Wang, Peilin Zhao, Steven CH Hoi, and Rong Jin · 2014
Cited alongside, same era.
Online decision-making with high-dimensional covariates
Hamsa Bastani and Mohsen Bayati · 2015
Cited alongside, same era.
Bandits and recommender systems
Jérémie Mary, Romaric Gaudel, and Philippe Preux · 2015
Cited alongside, same era.
Portfolio choices with orthogonal bandit learning
Weiwei Shen, Jun Wang, Yu-Gang Jiang, and Hongyuan Zha · 2015
Cited alongside, same era.
Multi-armed bandit problem with known trend
Djallel Bouneffouf and Raphaël Féraud · 2016
Cited alongside, same era.
Hyperband: A novel bandit-based approach to hyperparameter optimization
Original
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar · 2016
Cited alongside, same era.
Bandit models of human behavior: Reward processing in mental disorders
Djallel Bouneffouf, Irina Rish, and Guillermo A Cecchi · 2017
Cited alongside, same era.
Context attentive bandits: Contextual bandit with restricted context
Djallel Bouneffouf, Irina Rish, Guillermo A. Cecchi, and Raphaël Féraud · 2017
Cited alongside, same era.
Risk-aware multi-armed bandit problem with application to portfolio selection
Xiaoguang Huo and Feng Fu · 2017
Cited alongside, same era.
Algorithm selection of off-policy reinforcement learning algorithm
Original
Romain Laroche and Raphaël Féraud · 2017
Cited alongside, same era.
Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems
David E Losada, Javier Parapar, and Alvaro Barreiro · 2017
Cited alongside, same era.
Model-independent online learning for influence maximization
Sharan Vaswani, Branislav Kveton, Zheng Wen, Mohammad Ghavamzadeh, Laks VS Lakshmanan, and Mark Schmidt · 2017
Cited alongside, same era.

Then

Large-scale bandit approaches for recommender systems
Original
Qian Zhou, XiaoFang Zhang, Jin Xu, and Bin Liang · 2017
Later among the works it cites.
mumab: A multi-armed bandit model for wireless network selection
Stefano Boldrini, Luca De Nardis, Giuseppe Caso, Mai Le, Jocelyn Fiorina, and Maria-Gabriella Di Benedetto · 2018
Later among the works it cites.
Contextual bandits for adapting treatment in a mouse model of de novo carcinogenesis
Audrey Durand, Charis Achilleos, Demetris Iacovides, Katerina Strati, Georgios D Mitsis, and Joelle Pineau · 2018
Later among the works it cites.
Node-based optimization of lora transmissions with multi-armed bandit algorithms
Raouf Kerkouche, Réda Alami, Raphaël Féraud, Nadège Varsier, and Patrick Maillé · 2018
Later among the works it cites.
Customized nonlinear bandits for online response selection in neural conversation models
Bing Liu, Tong Yu, Ian Lane, and Ole J. Mengshoel · 2018
Later among the works it cites.
Dynamic online pricing with incomplete information using multi-armed bandit experiments
Kanishka Misra, Eric M Schwartz, and Jacob Abernethy · 2018
Later among the works it cites.
Low-rank bandit methods for high-dimensional dynamic pricing
Original
Jonas Mueller, Vasilis Syrgkanis, and Matt Taddy · 2018
Later among the works it cites.
Interpretable multi-objective reinforcement learning through policy orchestration
Original
Ritesh Noothigattu, Djallel Bouneffouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi · 2018
Later among the works it cites.
Contextual memory bandit for pro-active dialog engagement
Tomi Silander et al · 2018
Later among the works it cites.
Adapting to concept drift in credit card transaction data streams using contextual bandits and decision trees
Dennis JNJ Soemers, Tim Brys, Kurt Driessens, Mark HM Winands, and Ann Nowé · 2018
Later among the works it cites.
Collaborative clustering through constrained networks using bandit optimization
Jérémie Sublime and Sylvain Lefebvre · 2018
Later among the works it cites.
A bandit approach to posterior dialog orchestration under a budget
Sohini Upadhyay, Mayank Agarwal, Djallel Bounneffouf, and Yasaman Khazaeni · 2018
Later among the works it cites.
Interactive anomaly detection on attributed networks
Kaize Ding, Jundong Li, and Huan Liu · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…