Optimization-based Block Coordinate Gradient Coding

09/18/2021
by   Qi Wang, et al.
0

Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation system consisting of one master and N workers characterized by a general partial straggler model and focuses on solving a general large-scale machine learning problem with L model parameters. We show that it is sufficient to provide at most N levels of redundancies for tolerating 0, 1,⋯, N-1 stragglers, respectively. Consequently, we propose an optimal block coordinate gradient coding scheme based on a stochastic optimization problem that optimizes the partition of the L coordinates into N blocks, each with identical redundancy, to minimize the expected overall runtime for collaboratively computing the gradient. We obtain an optimal solution using a stochastic projected subgradient method and propose two low-complexity approximate solutions with closed-from expressions, for the stochastic optimization problem. We also show that under a shifted-exponential distribution, for any L, the expected overall runtimes of the two approximate solutions and the minimum overall runtime have sub-linear multiplicative gaps in N. To the best of our knowledge, this is the first work that optimizes the redundancies of gradient coding introduced across the coordinates of gradients.

READ FULL TEXT
research
06/06/2022

Optimization-based Block Coordinate Gradient Coding for Mitigating Partial Stragglers in Distributed Learning

Gradient coding schemes effectively mitigate full stragglers in distribu...
research
10/26/2017

Gradient Sparsification for Communication-Efficient Distributed Optimization

Modern large scale machine learning applications require stochastic opti...
research
04/30/2019

Harmonic Coding: An Optimal Linear Code for Privacy-Preserving Gradient-Type Computation

We consider the problem of distributedly computing a general class of fu...
research
02/19/2021

On Gradient Coding with Partial Recovery

We consider a generalization of the recently proposed gradient coding fr...
research
03/02/2021

Optimal Communication-Computation Trade-Off in Heterogeneous Gradient Coding

Gradient coding allows a master node to derive the aggregate of the part...
research
05/13/2021

Approximate Gradient Coding for Heterogeneous Nodes

In distributed machine learning (DML), the training data is distributed ...
research
01/28/2022

Measure Estimation in the Barycentric Coding Model

This paper considers the problem of measure estimation under the barycen...

Please sign up or login with your details

Forgot password? Click here to reset