Intrinsic Social Motivation via Causal Influence in Multi-Agent RL

by   Natasha Jaques, et al.

We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent's actions. Causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent's reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents' actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. We therefore employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.


page 5

page 7

page 13

page 16


Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

We propose a unified mechanism for achieving coordination and communicat...

Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication

We consider the problem of robust multi-agent reinforcement learning (MA...

Social navigation with human empowerment driven reinforcement learning

The next generation of mobile robots needs to be socially-compliant to b...

Reliably Re-Acting to Partner's Actions with the Social Intrinsic Motivation of Transfer Empowerment

We consider multi-agent reinforcement learning (MARL) for cooperative co...

Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

Can an arbitrarily intelligent reinforcement learning agent be kept unde...

Causal Influences over Social Learning Networks

This paper investigates causal influences between agents linked by a soc...

Discovering Agents

Causal models of agents have been used to analyse the safety aspects of ...

Please sign up or login with your details

Forgot password? Click here to reset