Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

by   Rui Wang, et al.

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.


page 7

page 8

page 20


Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions

While the history of machine learning so far encompasses a series of pro...

Open-ended search for environments and adapted agents using MAP-Elites

Creatures in the real world constantly encounter new and diverse challen...

A Survey of Demonstration Learning

With the fast improvement of machine learning, reinforcement learning (R...

Challenges and Governance Solutions for Data Science Services based on Open Data and APIs

Increasingly common open data and open application programming interface...

External control of a genetic toggle switch via Reinforcement Learning

We investigate the problem of using a learning-based strategy to stabili...

On the Reliability and Generalizability of Brain-inspired Reinforcement Learning Algorithms

Although deep RL models have shown a great potential for solving various...

Segment Anything in Non-Euclidean Domains: Challenges and Opportunities

The recent work known as Segment Anything (SA) has made significant stri...

Please sign up or login with your details

Forgot password? Click here to reset