Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery

by   Félix Chalumeau, et al.

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. However, these policies tend to be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training a mixture of policies, as opposed to a single one, that are driven to explore different regions of the state-action space can address this shortcoming by generating a diverse set of behaviors, referred to as skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including a diversity term - often derived from information theory - in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning to be effective. In this work, we demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are a competitive alternative to information-theory-augmented RL for skill discovery. Through an extensive empirical evaluation comparing eight state-of-the-art methods on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning; QD methods are found to provide equal, and sometimes improved, performance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide near-optimal performance across all environments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.


page 3

page 6

page 22

page 24

page 27

page 28

page 36


Discovering Generalizable Skills via Automated Generation of Diverse Tasks

The learning efficiency and generalization ability of an intelligent age...

Granger-Causal Hierarchical Skill Discovery

Reinforcement Learning (RL) has shown promising results learning policie...

The Information Geometry of Unsupervised Reinforcement Learning

How can a reinforcement learning (RL) agent prepare to solve downstream ...

Automata Guided Hierarchical Reinforcement Learning for Zero-shot Skill Composition

An obstacle that prevents the wide adoption of (deep) reinforcement lear...

Skill Decision Transformer

Recent work has shown that Large Language Models (LLMs) can be incredibl...

Hierarchical Reinforcement Learning for Sequencing Behaviors

Recent literature in the robot learning community has focused on learnin...

Evolving Populations of Diverse RL Agents with MAP-Elites

Quality Diversity (QD) has emerged as a powerful alternative optimizatio...

Please sign up or login with your details

Forgot password? Click here to reset