Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

by   Kaiqing Zhang, et al.

We consider the problem of fully decentralized multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.


Multi-Agent Decentralized Belief Propagation on Graphs

We consider the problem of interactive partially observable Markov decis...

Value Propagation for Decentralized Networked Deep Multi-agent Reinforcement Learning

We consider the networked multi-agent reinforcement learning (MARL) prob...

Finite-Sample Analyses for Fully Decentralized Multi-Agent Reinforcement Learning

Despite the increasing interest in multi-agent reinforcement learning (M...

Multi-agent Natural Actor-critic Reinforcement Learning Algorithms

Both single-agent and multi-agent actor-critic algorithms are an importa...

Provably Efficient Multi-Agent Reinforcement Learning with Fully Decentralized Communication

A challenge in reinforcement learning (RL) is minimizing the cost of sam...

Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Multi-agent reinforcement learning (MARL) has attracted much research at...

Cooperative Actor-Critic via TD Error Aggregation

In decentralized cooperative multi-agent reinforcement learning, agents ...

Please sign up or login with your details

Forgot password? Click here to reset