Communication Topologies Between Learning Agents in Deep Reinforcement Learning

by   Dhaval Adjodah, et al.

A common technique to improve speed and robustness of learning in deep reinforcement learning (DRL) and many other machine learning algorithms is to run multiple learning agents in parallel. A neglected component in the development of these algorithms has been how best to arrange the learning agents involved to better facilitate distributed search. Here we draw upon results from the networked optimization and collective intelligence literatures suggesting that arranging learning agents in less than fully connected topologies (the implicit way agents are commonly arranged in) can improve learning. We explore the relative performance of four popular families of graphs and observe that one such family (Erdos-Renyi random graphs) empirically outperforms the standard fully-connected communication topology across several DRL benchmark tasks. We observe that 1000 learning agents arranged in an Erdos-Renyi graph can perform as well as 3000 agents arranged in the standard fully-connected topology, showing the large learning improvement possible when carefully designing the topology over which agents communicate. We complement these empirical results with a preliminary theoretical investigation of why less than fully connected topologies can perform better. Overall, our work suggests that distributed machine learning algorithms could be made more efficient if the communication topology between learning agents was optimized.


page 1

page 2

page 3

page 4


How to Organize your Deep Reinforcement Learning Agents: The Importance of Communication Topology

In this empirical paper, we investigate how learning agents can be arran...

An Empirical Deep Dive into Deep Learning's Driving Dynamics

We present an empirical dataset surveying the deep learning phenomenon o...

Improved Learning in Evolution Strategies via Sparser Inter-Agent Network Topologies

We draw upon a previously largely untapped literature on human collectiv...

Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS

The human cultural repertoire relies on innovation: our ability to conti...

Connectivity-driven Communication in Multi-agent Reinforcement Learning through Diffusion Processes on Graphs

We discuss the problem of learning collaborative behaviour in multi-agen...

Dynamic communication topologies for distributed heuristics in energy system optimization algorithms

The communication topology is an essential aspect in designing distribut...

Featuring the topology with the unsupervised machine learning

Images of line drawings are generally composed of primitive elements. On...

Please sign up or login with your details

Forgot password? Click here to reset