Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains

by   Manon Flageat, et al.

Quality-Diversity algorithms, among which MAP-Elites, have emerged as powerful alternatives to performance-only optimisation approaches as they enable generating collections of diverse and high-performing solutions to an optimisation problem. However, they are often limited to low-dimensional search spaces and deterministic environments. The recently introduced Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) algorithm overcomes this limitation by pairing the traditional Genetic operator of MAP-Elites with a gradient-based operator inspired by Deep Reinforcement Learning. This new operator guides mutations toward high-performing solutions using policy-gradients. In this work, we propose an in-depth study of PGA-MAP-Elites. We demonstrate the benefits of policy-gradients on the performance of the algorithm and the reproducibility of the generated solutions when considering uncertain domains. We first prove that PGA-MAP-Elites is highly performant in both deterministic and uncertain high-dimensional environments, decorrelating the two challenges it tackles. Secondly, we show that in addition to outperforming all the considered baselines, the collections of solutions generated by PGA-MAP-Elites are highly reproducible in uncertain environments, approaching the reproducibility of solutions found by Quality-Diversity approaches built specifically for uncertain applications. Finally, we propose an ablation and in-depth analysis of the dynamic of the policy-gradients-based variation. We demonstrate that the policy-gradient variation operator is determinant to guarantee the performance of PGA-MAP-Elites but is only essential during the early stage of the process, where it finds high-performing regions of the search space.


page 1

page 3

page 13

page 14

page 25

page 26


MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy

Quality-Diversity algorithms, such as MAP-Elites, are a branch of Evolut...

Improving the Data Efficiency of Multi-Objective Quality-Diversity through Gradient Assistance and Crowding Exploration

Quality-Diversity (QD) algorithms have recently gained traction as optim...

Benchmark tasks for Quality-Diversity applied to Uncertain domains

While standard approaches to optimisation focus on producing a single hi...

Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains

Quality-Diversity (QD) algorithms are designed to generate collections o...

Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution

Neuroevolution is an alternative to gradient-based optimisation that has...

Uncertain Quality-Diversity: Evaluation methodology and new methods for Quality-Diversity in Uncertain Domains

Quality-Diversity optimisation (QD) has proven to yield promising result...

Discovering the Elite Hypervolume by Leveraging Interspecies Correlation

Evolution has produced an astonishing diversity of species, each filling...

Please sign up or login with your details

Forgot password? Click here to reset