Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning

by   Sohei Itahara, et al.

Federated learning (FL) enables a neural network (NN) to be trained using privacy-sensitive data on mobile devices while retaining all the data on their local storages. However, FL asks the mobile devices to perform heavy communication and computation tasks, i.e., devices are requested to upload and download large-volume NN models and train them. This paper proposes a novel unsupervised pre-training method adapted for FL, which aims to reduce both the communication and computation costs through model compression. Since the communication and computation costs are highly dependent on the volume of NN models, reducing the volume without decreasing model performance can reduce these costs. The proposed pre-training method leverages unlabeled data, which is expected to be obtained from the Internet or data repository much more easily than labeled data. The key idea of the proposed method is to obtain a “good” subnetwork from the original NN using the unlabeled data based on the lottery hypothesis. The proposed method trains an original model using a denoising auto encoder with the unlabeled data and then prunes small-magnitude parameters of the original model to generate a small but good subnetwork. The proposed method is evaluated using an image classification task. The results show that the proposed method requires 35% less traffic and computation time than previous methods when achieving a certain test accuracy.


Estimation of Individual Device Contributions for Incentivizing Federated Learning

Federated learning (FL) is an emerging technique used to train a machine...

Low-Parameter Federated Learning with Large Language Models

We study few-shot Natural Language Understanding (NLU) tasks with Large ...

CoCo-FL: Communication- and Computation-Aware Federated Learning via Partial NN Freezing and Quantization

Devices participating in federated learning (FL) typically have heteroge...

Federated Contrastive Learning for Dermatological Disease Diagnosis via On-device Learning

Deep learning models have been deployed in an increasing number of edge ...

Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data

This study develops a federated learning (FL) framework overcoming large...

Heterogeneous Data-Aware Federated Learning

Federated learning (FL) is an appealing concept to perform distributed t...

Please sign up or login with your details

Forgot password? Click here to reset