Rethinking Client Drift in Federated Learning: A Logit Perspective

by   Yunlu Yan, et al.

Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. However, the real-world non-IID data will lead to client drift which degrades the performance of FL. Interestingly, we find that the difference in logits between the local and global models increases as the model is continuously updated, thus seriously deteriorating FL performance. This is mainly due to catastrophic forgetting caused by data heterogeneity between clients. To alleviate this problem, we propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models. FedCSD does not simply transfer global knowledge to local clients, as an undertrained global model cannot provide reliable knowledge, i.e., class similarity information, and its wrong soft labels will mislead the optimization of local models. Concretely, FedCSD introduces a class prototype similarity distillation to align the local logits with the refined global logits that are weighted by the similarity between local logits and the global prototype. To enhance the quality of global logits, FedCSD adopts an adaptive mask to filter out the terrible soft labels of the global models, thereby preventing them to mislead local optimization. Extensive experiments demonstrate the superiority of our method over the state-of-the-art federated learning approaches in various heterogeneous settings. The source code will be released.


page 1

page 9


Federated Learning on Heterogeneous Data via Adaptive Self-Distillation

Federated Learning (FL) is a machine learning paradigm that enables clie...

Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Despite Federated Learning (FL)'s trend for learning machine learning mo...

Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape

In federated learning (FL), a cluster of local clients are chaired under...

HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images

Multiple medical institutions collaboratively training a model using fed...

FedSiam-DA: Dual-aggregated Federated Learning via Siamese Network under Non-IID Data

Federated learning is a distributed learning that allows each client to ...

Knowledge-Aware Federated Active Learning with Non-IID Data

Federated learning enables multiple decentralized clients to learn colla...

Federated Learning with Client-Exclusive Classes

Existing federated classification algorithms typically assume the local ...

Please sign up or login with your details

Forgot password? Click here to reset