Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning

by   R. Srikant, et al.

We consider the dynamics of a linear stochastic approximation algorithm driven by Markovian noise, and derive finite-time bounds on the moments of the error, i.e., deviation of the output of the algorithm from the equilibrium point of an associated ordinary differential equation (ODE). To obtain finite-time bounds on the mean-square error in the case of constant step-size algorithms, our analysis uses Stein's method to identify a Lyapunov function that can potentially yield good steady-state bounds, and uses this Lyapunov function to obtain finite-time bounds by mimicking the corresponding steps in the analysis of the associated ODE. We also provide a comprehensive treatment of the moments of the square of the 2-norm of the approximation error. Our analysis yields the following results: (i) for a given step-size, we show that the lower-order moments can be made small as a function of the step-size and can be upper-bounded by the moments of a Gaussian random variable; (ii) we show that the higher-order moments beyond a threshold may be infinite in steady-state; and (iii) we characterize the number of samples needed for the finite-time bounds to be of the same order as the steady-state bounds. As a by-product of our analysis, we also solve the open problem of obtaining finite-time bounds for the performance of temporal difference learning algorithms with linear function approximation and a constant step-size, without requiring a projection step or an i.i.d. noise assumption.


page 1

page 2

page 3

page 4


Finite-Time Error Bounds for Distributed Linear Stochastic Approximation

This paper considers a novel multi-agent linear stochastic approximation...

Finite-Time Analysis of Q-Learning with Linear Function Approximation

In this paper, we consider the model-free reinforcement learning problem...

Error bound analysis of the stochastic parareal algorithm

Stochastic parareal (SParareal) is a probabilistic variant of the popula...

On the Correlation between the Noise and a Priori Error Vectors in Affine Projection Algorithms

This paper analyzes the correlation matrix between the a priori error an...

Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise

Linear two-timescale stochastic approximation (SA) scheme is an importan...

Stochastic Learning under Random Reshuffling

In empirical risk optimization, it has been observed that stochastic gra...

Tracking Performance of Online Stochastic Learners

The utilization of online stochastic algorithms is popular in large-scal...

Please sign up or login with your details

Forgot password? Click here to reset