Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization

by   Wei Jiang, et al.

Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations. What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of 𝒪(1) functional mappings at each iteration? There is an important application in solving an emerging family of coupled compositional optimization problems in the form of ∑_i=1^m f_i(g_i(𝐰)), where g_i is accessible through a stochastic oracle. The key issue is to track and estimate a sequence of 𝐠(𝐰)=(g_1(𝐰), …, g_m(𝐰)) across iterations, where 𝐠(𝐰) has m blocks and it is only allowed to probe 𝒪(1) blocks to attain their stochastic values and Jacobians. To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Single-probe Variance Reduced (MSVR) estimator to track the sequence of 𝐠(𝐰). It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled. With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex objectives. Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter. Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator.


page 1

page 2

page 3

page 4


Optimal Algorithms for Stochastic Multi-Level Compositional Optimization

In this paper, we investigate the problem of stochastic multi-level comp...

Multi-block Min-max Bilevel Optimization with Applications in Multi-task Deep AUC Maximization

In this paper, we study multi-block min-max bilevel optimization problem...

Projection-Free Algorithm for Stochastic Bi-level Optimization

This work presents the first projection-free algorithm to solve stochast...

Finite-Sum Compositional Stochastic Optimization: Theory and Applications

This paper studies stochastic optimization for a sum of compositional fu...

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

In this paper, we consider non-convex multi-block bilevel optimization (...

A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequences

Stochastic approximation (SA) with multiple coupled sequences has found ...

Provable Multi-instance Deep AUC Maximization with Stochastic Pooling

This paper considers a novel application of deep AUC maximization (DAM) ...

Please sign up or login with your details

Forgot password? Click here to reset