UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers

by   Siyi Hu, et al.

Recent advances in multi-agent reinforcement learning have been largely limited in training one model from scratch for every new task. The limitation is due to the restricted model architecture related to fixed input and output dimensions. This hinders the experience accumulation and transfer of the learned agent over tasks with diverse levels of difficulty (e.g. 3 vs 3 or 5 vs 6 multi-agent games). In this paper, we make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks with the requirement of different observation and action configurations. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy by decoupling the policy distribution from the intertwined input observation with an importance weight measured by the merits of the self-attention mechanism. Compared to a standard transformer block, the proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable. UPDeT is general enough to be plugged into any multi-agent reinforcement learning pipeline and equip them with strong generalization abilities that enables the handling of multiple tasks at a time. Extensive experiments on large-scale SMAC multi-agent competitive games demonstrate that the proposed UPDeT-based multi-agent reinforcement learning achieves significant results relative to state-of-the-art approaches, demonstrating advantageous transfer capability in terms of both performance and training speed (10 times faster).


Deep Multi-Agent Reinforcement Learning with Relevance Graphs

Over recent years, deep reinforcement learning has shown strong successe...

Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning

Policy optimization methods with function approximation are widely used ...

Efficient Domain Coverage for Vehicles with Second Order Dynamics via Multi-Agent Reinforcement Learning

Collaborative autonomous multi-agent systems covering a specified area h...

Explainable Action Advising for Multi-Agent Reinforcement Learning

Action advising is a knowledge transfer technique for reinforcement lear...

Concept Learning for Interpretable Multi-Agent Reinforcement Learning

Multi-agent robotic systems are increasingly operating in real-world env...

EMOTE: An Explainable architecture for Modelling the Other Through Empathy

We can usually assume others have goals analogous to our own. This assum...

Automating Turbulence Modeling by Multi-Agent Reinforcement Learning

The modeling of turbulent flows is critical to scientific and engineerin...

Please sign up or login with your details

Forgot password? Click here to reset