Online Learning and Optimization of Markov Jump Affine Models

Understand

The problem of online learning and optimization of unknown Markov jump affine models is considered.

• An online learning policy, referred to as Markovian simultaneous perturbations stochastic approximation (MSPSA), is proposed for two different optimization objectives: (i) the quadratic cost minimization of the regulation problem and (ii) the revenue (profit) maximization problem.
• It is shown that the regret of MSPSA grows at the order of the square root of the learning horizon.
• Furthermore, by the use of van Trees inequality, it is shown that the regret of any policy grows no slower than that of MSPSA, making MSPSA an order optimal learning policy.