Adaptive Stochastic Gradient Langevin Dynamics: Taming Convergence and Saddle Point Escape Time
share
In this paper, we propose a new adaptive stochastic gradient Langevin dynamics (ASGLD) algorithmic framework and its two specialized versions, namely adaptive stochastic gradient (ASG) and adaptive gradient Langevin dynamics(AGLD), for non-convex optimization problems. All proposed algorithms can escape from saddle points with at most O( d) iterations, which is nearly dimension-free. Further, we show that ASGLD and ASG converge to a local minimum with at most O( d/ϵ^4) iterations. Also, ASGLD with full gradients or ASGLD with a slowly linearly increasing batch size converge to a local minimum with iterations bounded by O( d/ϵ^2), which outperforms existing first-order methods.
READ FULL TEXT