Federated Transfer Learning with Dynamic Gradient Aggregation

by   Dimitrios Dimitriadis, et al.

In this paper, a Federated Learning (FL) simulation platform is introduced. The target scenario is Acoustic Model training based on this platform. To our knowledge, this is the first attempt to apply FL techniques to Speech Recognition tasks due to the inherent complexity. The proposed FL platform can support different tasks based on the adopted modular design. As part of the platform, a novel hierarchical optimization scheme and two gradient aggregation methods are proposed, leading to almost an order of magnitude improvement in training convergence speed compared to other distributed or FL training algorithms like BMUF and FedAvg. The hierarchical optimization offers additional flexibility in the training pipeline besides the enhanced convergence speed. On top of the hierarchical optimization, a dynamic gradient aggregation algorithm is proposed, based on a data-driven weight inference. This aggregation algorithm acts as a regularizer of the gradient quality. Finally, an unsupervised training pipeline tailored to FL is presented as a separate training scenario. The experimental validation of the proposed system is based on two tasks: first, the LibriSpeech task showing a speed-up of 7x and 6 second task is based on session adaptation providing an improvement of 20 over a competitive production-ready LAS model. The proposed Federated Learning system is shown to outperform the golden standard of distributed training in both convergence speed and overall model performance.


page 1

page 2

page 3

page 4


Dynamic Gradient Aggregation for Federated Domain Adaptation

In this paper, a new learning algorithm for Federated Learning (FL) is i...

Hierarchical Federated Learning through LAN-WAN Orchestration

Federated learning (FL) was designed to enable mobile phones to collabor...

Towards More Efficient Federated Learning with Better Optimization Objects

Federated Learning (FL) is a privacy-protected machine learning paradigm...

Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating

Federated Averaging (FedAvg) serves as the fundamental framework in Fede...

AVDDPG: Federated reinforcement learning applied to autonomous platoon control

Since 2016 federated learning (FL) has been an evolving topic of discuss...

Covert Model Poisoning Against Federated Learning: Algorithm Design and Optimization

Federated learning (FL), as a type of distributed machine learning frame...

SuperFed: Weight Shared Federated Learning

Federated Learning (FL) is a well-established technique for privacy pres...

Please sign up or login with your details

Forgot password? Click here to reset