Automated curricula through setter-solver interactions

by   Sebastien Racanière, et al.

Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic or sparsely rewarding environments these correlations are often too small, or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula–the breakdown of tasks into simpler, static challenges with dense rewards–to build up to complex behaviors. While curricula are also useful for artificial agents, hand-crafting them is time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.


Stein Variational Goal Generation For Reinforcement Learning in Hard Exploration Problems

Multi-goal Reinforcement Learning has recently attracted a large amount ...

Goal-Conditioned Q-Learning as Knowledge Distillation

Many applications of reinforcement learning can be formalized as goal-co...

It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation

We are interested in training general-purpose reinforcement learning age...

Assessing Evolutionary Terrain Generation Methods for Curriculum Reinforcement Learning

Curriculum learning allows complex tasks to be mastered via incremental ...

Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft

An important challenge in reinforcement learning is training agents that...

Meta Automatic Curriculum Learning

A major challenge in the Deep RL (DRL) community is to train agents able...

Learning Universal Policies via Text-Guided Video Generation

A goal of artificial intelligence is to construct an agent that can solv...

Please sign up or login with your details

Forgot password? Click here to reset