Policy Gradient Optimization of Thompson Sampling Policies

Understand

We study the use of policy gradient algorithms to optimize over a class of generalized Thompson sampling policies.

• Our central insight is to view the posterior parameter sampled by Thompson sampling as a kind of pseudo-action.
• Policy gradient methods can then be tractably applied to search over a class of sampling policies, which determine a probability distribution over pseudo-actions (i.e., sampled parameters) as a function of observed data.
• We also propose and compare policy gradient estimators that are specialized to Bayesian bandit problems.