Stochastic Controlled Averaging for Federated Learning with Communication Compression

by   Xinmeng Huang, et al.

Communication compression, a technique aiming to reduce the information volume to be transmitted over the air, has gained great interests in Federated Learning (FL) for the potential of alleviating its communication overhead. However, communication compression brings forth new challenges in FL due to the interplay of compression-incurred information distortion and inherent characteristics of FL such as partial participation and data heterogeneity. Despite the recent development, the performance of compressed FL approaches has not been fully exploited. The existing approaches either cannot accommodate arbitrary data heterogeneity or partial participation, or require stringent conditions on compression. In this paper, we revisit the seminal stochastic controlled averaging method by proposing an equivalent but more efficient/simplified formulation with halved uplink communication costs. Building upon this implementation, we propose two compressed FL algorithms, SCALLION and SCAFCOM, to support unbiased and biased compression, respectively. Both the proposed methods outperform the existing compressed FL methods in terms of communication and computation complexities. Moreover, SCALLION and SCAFCOM accommodates arbitrary data heterogeneity and do not make any additional assumptions on compression errors. Experiments show that SCALLION and SCAFCOM can match the performance of corresponding full-precision FL approaches with substantially reduced uplink communication, and outperform recent compressed FL methods under the same communication budget.


page 1

page 2

page 3

page 4


Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates

Federated Learning (FL) is an emerging decentralized learning framework ...

QUIC-FL: Quick Unbiased Compression for Federated Learning

Distributed Mean Estimation (DME) is a fundamental building block in com...

Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates

Federated Learning (FL) solves many of this decade's concerns regarding ...

Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression

In federated learning (FL) systems, e.g., wireless networks, the communi...

HFedMS: Heterogeneous Federated Learning with Memorable Data Semantics in Industrial Metaverse

Federated Learning (FL), as a rapidly evolving privacy-preserving collab...

Compressed Particle-Based Federated Bayesian Learning and Unlearning

Conventional frequentist FL schemes are known to yield overconfident dec...

Federated Learning over Noisy Channels: Convergence Analysis and Design Examples

Does Federated Learning (FL) work when both uplink and downlink communic...

Please sign up or login with your details

Forgot password? Click here to reset