Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization

by   Qi Deng, et al.

Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradient method that randomly updates block-coordinate variables by using block composite subgradient mapping. This method converges asymptotically to critical points with proven sublinear convergence rate for certain optimality measures. Second, we develop a randomly permuted CD method with two alternating steps: linearizing the concave part and cycling through variables. We prove asymptotic convergence to critical points and sublinear complexity rate for objectives with both smooth and concave parts. Third, we extend accelerated coordinate descent (ACD) to nonsmooth and nonconvex optimization to develop a novel randomized proximal DC algorithm whereby we solve the subproblem inexactly by ACD. Convergence is guaranteed with at most a few number of ACD iterations for each DC subproblem, and convergence complexity is established for identification of some approximate critical points. Fourth, we further develop the third method to minimize certain ill-conditioned nonconvex functions: weakly convex functions with high Lipschitz constant to negative curvature ratios. We show that, under specific criteria, the ACD-based randomized method has superior complexity compared to conventional gradient methods. Finally, an empirical study on sparsity-inducing learning models demonstrates that CD methods are superior to gradient-based methods for certain large-scale problems.


page 1

page 2

page 3

page 4


A Randomized Nonmonotone Block Proximal Gradient Method for a Class of Structured Nonlinear Programming

We propose a randomized nonmonotone block proximal gradient (RNBPG) meth...

Coordinate Descent Methods for DC Minimization

Difference-of-Convex (DC) minimization, referring to the problem of mini...

Coordinate Descent Methods for Fractional Minimization

We consider a class of structured fractional minimization problems, in w...

Randomized Coordinate Subgradient Method for Nonsmooth Optimization

Nonsmooth optimization finds wide applications in many engineering field...

Iteration Complexity of Randomized Block-Coordinate Descent Methods for Minimizing a Composite Function

In this paper we develop a randomized block-coordinate descent method fo...

Complexity of Block Coordinate Descent with Proximal Regularization and Applications to Wasserstein CP-dictionary Learning

We consider the block coordinate descent methods of Gauss-Seidel type wi...

A Block Coordinate Descent Method for Nonsmooth Composite Optimization under Orthogonality Constraints

Nonsmooth composite optimization with orthogonality constraints has a br...

Please sign up or login with your details

Forgot password? Click here to reset