Homotopy Smoothing for Non-Smooth Problems with Lower Complexity than O(1/ε)

by   Yi Xu, et al.

In this paper, we develop a novel homoto py smoothing (HOPS) algorithm for solving a family of non-smooth problems that is composed of a non-smooth term with an explicit max-structure and a smooth term or a simple non-smooth term whose proximal mapping is easy to compute. The best known iteration complexity for solving such non-smooth optimization problems is O(1/ϵ) without any assumption on the strong convexity. In this work, we will show that the proposed HOPS achieved a lower iteration complexity of O(1/ϵ^1-θ)[ O() suppresses a logarithmic factor.] with θ∈(0,1] capturing the local sharpness of the objective function around the optimal solutions. To the best of our knowledge, this is the lowest iteration complexity achieved so far for the considered non-smooth optimization problems without strong convexity assumption. The HOPS algorithm employs Nesterov's smoothing technique and Nesterov's accelerated gradient method and runs in stages, which gradually decreases the smoothing parameter in a stage-wise manner until it yields a sufficiently good approximation of the original function. We show that HOPS enjoys a linear convergence for many well-known non-smooth problems (e.g., empirical risk minimization with a piece-wise linear loss function and ℓ_1 norm regularizer, finding a point in a polyhedron, cone programming, etc). Experimental results verify the effectiveness of HOPS in comparison with Nesterov's smoothing algorithm and the primal-dual style of first-order methods.


page 1

page 2

page 3

page 4


Solving Non-smooth Constrained Programs with Lower Complexity than O(1/ε): A Primal-Dual Homotopy Smoothing Approach

We propose a new primal-dual homotopy smoothing algorithm for a linearly...

Lifted Primal-Dual Method for Bilinearly Coupled Smooth Minimax Optimization

We study the bilinearly coupled minimax problem: min_xmax_y f(x) + y^⊤ A...

Adaptive Smoothing Algorithms for Nonsmooth Composite Convex Minimization

We propose an adaptive smoothing algorithm based on Nesterov's smoothing...

Stochastic proximal splitting algorithm for stochastic composite minimization

Supported by the recent contributions in multiple branches, the first-or...

Higher-Order Accelerated Methods for Faster Non-Smooth Optimization

We provide improved convergence rates for various non-smooth optimizatio...

Oracle Complexity in Nonsmooth Nonconvex Optimization

It is well-known that given a smooth, bounded-from-below, and possibly n...

Efficient MRF Energy Minimization via Adaptive Diminishing Smoothing

We consider the linear programming relaxation of an energy minimization ...

Please sign up or login with your details

Forgot password? Click here to reset