How To Make the Gradients Small Stochastically

01/08/2018

∙

In convex stochastic optimization, convergence rates in terms of minimizing the objective have been well-established. However, in terms of making the gradients small, the best known convergence rate was O(ε^-8/3) and it was left open how to improve it. In this paper, we improve this rate to Õ(ε^-2), which is optimal up to log factors.

READ FULL TEXT

How To Make the Gradients Small Stochastically

Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication

Convergence Rates for Projective Splitting

Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions

Fundamental limits of over-the-air optimization: Are analog schemes optimal?

Convergence Rates for Non-Log-Concave Sampling and Log-Partition Estimation

Finite Precision Stochastic Optimization -- Accounting for the Bias

How To Make the Gradients Small Stochastically

Related Research

Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuity

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication

Convergence Rates for Projective Splitting

Stochastic optimization and sparse statistical recovery: An optimal algorithm for high dimensions

Fundamental limits of over-the-air optimization: Are analog schemes optimal?

Convergence Rates for Non-Log-Concave Sampling and Log-Partition Estimation

Finite Precision Stochastic Optimization -- Accounting for the Bias