Sparse Allreduce: Efficient Scalable Communication for Power-Law Data

by   Huasha Zhao, et al.

Many large datasets exhibit power-law statistics: The web graph, social networks, text data, click through data etc. Their adjacency graphs are termed natural graphs, and are known to be difficult to partition. As a consequence most distributed algorithms on these graphs are communication intensive. Many algorithms on natural graphs involve an Allreduce: a sum or average of partitioned data which is then shared back to the cluster nodes. Examples include PageRank, spectral partitioning, and many machine learning algorithms including regression, factor (topic) models, and clustering. In this paper we describe an efficient and scalable Allreduce primitive for power-law data. We point out scaling problems with existing butterfly and round-robin networks for Sparse Allreduce, and show that a hybrid approach improves on both. Furthermore, we show that Sparse Allreduce stages should be nested instead of cascaded (as in the dense case). And that the optimum throughput Allreduce network should be a butterfly of heterogeneous degree where degree decreases with depth into the network. Finally, a simple replication scheme is introduced to deal with node failures. We present experiments showing significant improvements over existing systems such as PowerGraph and Hadoop.


page 1

page 2

page 3

page 4


DRONE: a Distributed Subgraph-Centric Framework for Processing Large Scale Power-law Graphs

Nowadays, in the big data era, social networks, graph databases, knowled...

Graph Partitioning via Parallel Submodular Approximation to Accelerate Distributed Machine Learning

Distributed computing excels at processing large scale data, but the com...

An Efficient and Balanced Graph Partition Algorithm for the Subgraph-Centric Programming Model on Large-scale Power-law Graphs

The subgraph-centric programming model is a promising approach and has b...

DRONE: a Distributed gRaph cOmputiNg Engine

Nowadays, in big data era, social networks, graph database, knowledge gr...

A Dynamic Process Reference Model for Sparse Networks with Reciprocity

Many social and other networks exhibit stable size scaling relationships...

Superconducting Optoelectronic Neurons V: Networks and Scaling

Networks of superconducting optoelectronic neurons are investigated for ...

Adapting Stochastic Block Models to Power-Law Degree Distributions

Stochastic block models (SBMs) have been playing an important role in mo...

Please sign up or login with your details

Forgot password? Click here to reset