Kiefer Wolfowitz Algorithm is Asymptotically Optimal for a Class of Non-Stationary Bandit Problems
share
We consider the problem of designing an allocation rule or an "online learning algorithm" for a class of bandit problems in which the set of control actions available at each time s is a convex, compact subset of R^d. Upon choosing an action x at time s, the algorithm obtains a noisy value of the unknown and time-varying function f_s evaluated at x. The "regret" of an algorithm is the gap between its expected reward, and the reward earned by a strategy which has the knowledge of the function f_s at each time s and hence chooses the action x_s that maximizes f_s. For this non-stationary bandit problem set-up, we consider two variants of the Kiefer Wolfowitz (KW) algorithm i) KW with fixed step-size β, and ii) KW with sliding window of length L. We show that if the number of times that the function f_s varies during time T is o(T), and if the learning rates of the proposed algorithms are chosen "optimally", then the regret of the proposed algorithms is o(T), and hence the algorithms are asymptotically efficient.
READ FULL TEXT