Exploration in Deep Reinforcement Learning: A Comprehensive Survey

by   Tianpei Yang, et al.

Deep Reinforcement Learning (DRL) and Deep Multi-agent Reinforcement Learning (MARL) have achieved significant success across a wide range of domains, such as game AI, autonomous vehicles, robotics and finance. However, DRL and deep MARL agents are widely known to be sample-inefficient and millions of interactions are usually needed even for relatively simple game settings, thus preventing the wide application in real-industry scenarios. One bottleneck challenge behind is the well-known exploration problem, i.e., how to efficiently explore the unknown environments and collect informative experiences that could benefit the policy learning most. In this paper, we conduct a comprehensive survey on existing exploration methods in DRL and deep MARL for the purpose of providing understandings and insights on the critical problems and solutions. We first identify several key challenges to achieve efficient exploration, which most of the exploration methods aim at addressing. Then we provide a systematic survey of existing approaches by classifying them into two major categories: uncertainty-oriented exploration and intrinsic motivation-oriented exploration. The essence of uncertainty-oriented exploration is to leverage the quantification of the epistemic and aleatoric uncertainty to derive efficient exploration. By contrast, intrinsic motivation-oriented exploration methods usually incorporate different reward agnostic information for intrinsic exploration guidance. Beyond the above two main branches, we also conclude other exploration methods which adopt sophisticated techniques but are difficult to be classified into the above two categories. In addition, we provide a comprehensive empirical comparison of exploration methods for DRL on a set of commonly used benchmarks. Finally, we summarize the open problems of exploration in DRL and deep MARL and point out a few future directions.


page 1

page 5

page 6


Exploration in Deep Reinforcement Learning: A Survey

This paper reviews exploration techniques in deep reinforcement learning...

A Survey of Deep Reinforcement Learning in Video Games

Deep reinforcement learning (DRL) has made great achievements since prop...

Explore-Bench: Data Sets, Metrics and Evaluations for Frontier-based and Deep-reinforcement-learning-based Autonomous Exploration

Autonomous exploration and mapping of unknown terrains employing single ...

Principled Exploration via Optimistic Bootstrapping and Backward Induction

One principled approach for provably efficient exploration is incorporat...

A survey on intrinsic motivation in reinforcement learning

Despite numerous research work in reinforcement learning (RL) and the re...

Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey

Deep Reinforcement Learning (DRL) and Evolution Strategies (ESs) have su...

Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization

Recently deep reinforcement learning (DRL) has achieved outstanding succ...

Please sign up or login with your details

Forgot password? Click here to reset