Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Data

by   Ghadir Ayache, et al.

We consider the problem of a Parameter Server (PS) that wishes to learn a model that fits data distributed on the nodes of a graph. We focus on Federated Learning (FL) as a canonical application. One of the main challenges of FL is the communication bottleneck between the nodes and the parameter server. A popular solution in the literature is to allow each node to do several local updates on the model in each iteration before sending it back to the PS. While this mitigates the communication bottleneck, the statistical heterogeneity of the data owned by the different nodes has proven to delay convergence and bias the model. In this work, we study random walk (RW) learning algorithms for tackling the communication and data heterogeneity problems. The main idea is to leverage available direct connections among the nodes themselves, which are typically "cheaper" than the communication to the PS. In a random walk, the model is thought of as a "baton" that is passed from a node to one of its neighbors after being updated in each iteration. The challenge in designing the RW is the data heterogeneity and the uncertainty about the data distributions. Ideally, we would want to visit more often nodes that hold more informative data. We cast this problem as a sleeping multi-armed bandit (MAB) to design a near-optimal node sampling strategy that achieves variance-reduced gradient estimates and approaches sub-linearly the optimal sampling strategy. Based on this framework, we present an adaptive random walk learning algorithm. We provide theoretical guarantees on its convergence. Our numerical results validate our theoretical findings and show that our algorithm outperforms existing random walk algorithms.


Private Weighted Random Walk Stochastic Gradient Descent

We consider a decentralized learning setting in which data is distribute...

Mobilizing Personalized Federated Learning via Random Walk Stochastic ADMM

In this research, we investigate the barriers associated with implementi...

A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization

This paper addresses consensus optimization problem in a multi-agent net...

Het-node2vec: second order random walk sampling for heterogeneous multigraphs embedding

We introduce a set of algorithms (Het-node2vec) that extend the original...

Random Walk Bandits

Bandit learning problems find important applications ranging from medica...

FedWalk: Communication Efficient Federated Unsupervised Node Embedding with Differential Privacy

Node embedding aims to map nodes in the complex graph into low-dimension...

Dynamic Federated Learning

Federated learning has emerged as an umbrella term for centralized coord...

Please sign up or login with your details

Forgot password? Click here to reset