Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning

by   Yujun Shi, et al.

Federated learning aims to train models collaboratively across different clients without the sharing of data for privacy considerations. However, one major challenge for this learning paradigm is the data heterogeneity problem, which refers to the discrepancies between the local data distributions among various clients. To tackle this problem, we first study how data heterogeneity affects the representations of the globally aggregated models. Interestingly, we find that heterogeneous data results in the global model suffering from severe dimensional collapse, in which representations tend to reside in a lower-dimensional space instead of the ambient space. Moreover, we observe a similar phenomenon on models locally trained on each client and deduce that the dimensional collapse on the global model is inherited from local models. In addition, we theoretically analyze the gradient flow dynamics to shed light on how data heterogeneity result in dimensional collapse for local models. To remedy this problem caused by the data heterogeneity, we propose FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning. Specifically, FedDecorr applies a regularization term during local training that encourages different dimensions of representations to be uncorrelated. FedDecorr, which is implementation-friendly and computationally-efficient, yields consistent improvements over baselines on standard benchmark datasets. Code will be released.


page 1

page 2

page 3

page 4


Federated Learning in Temporal Heterogeneity

In this work, we explored federated learning in temporal heterogeneity a...

HetVis: A Visual Analysis Approach for Identifying Data Heterogeneity in Horizontal Federated Learning

Horizontal federated learning (HFL) enables distributed clients to train...

Investigating Neuron Disturbing in Fusing Heterogeneous Neural Networks

Fusing deep learning models trained on separately located clients into a...

DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup

Recently, federated learning has emerged as a promising approach for tra...

Tackling Data Heterogeneity in Federated Learning with Class Prototypes

Data heterogeneity across clients in federated learning (FL) settings is...

FedSiam-DA: Dual-aggregated Federated Learning via Siamese Network under Non-IID Data

Federated learning is a distributed learning that allows each client to ...

FedStack: Personalized activity monitoring using stacked federated learning

Recent advances in remote patient monitoring (RPM) systems can recognize...

Please sign up or login with your details

Forgot password? Click here to reset