DEHB: Evolutionary Hyberband for Scalable, Robust and Efficient Hyperparameter Optimization

by   Noor Awad, et al.

Modern machine learning algorithms crucially rely on several design decisions to achieve strong performance, making the problem of Hyperparameter Optimization (HPO) more important than ever. Here, we combine the advantages of the popular bandit-based HPO method Hyperband (HB) and the evolutionary search approach of Differential Evolution (DE) to yield a new HPO method which we call DEHB. Comprehensive results on a very broad range of HPO problems, as well as a wide range of tabular benchmarks from neural architecture search, demonstrate that DEHB achieves strong performance far more robustly than all previous HPO methods we are aware of, especially for high-dimensional problems with discrete input dimensions. For example, DEHB is up to 1000x faster than random search. It is also efficient in computational time, conceptually simple and easy to implement, positioning it well to become a new default HPO method.


BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Modern deep learning methods are very sensitive to many hyperparameters,...

MO-DEHB: Evolutionary-based Hyperband for Multi-Objective Optimization

Hyperparameter optimization (HPO) is a powerful technique for automating...

Weighted Random Search for Hyperparameter Optimization

We introduce an improved version of Random Search (RS), used here for hy...

Learning Multiple Defaults for Machine Learning Algorithms

The performance of modern machine learning methods highly depends on the...

Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges

Most machine learning algorithms are configured by one or several hyperp...

Evolutionary optimization of an experimental apparatus

In recent decades, cold atom experiments have become increasingly comple...

Tune: A Research Platform for Distributed Model Selection and Training

Modern machine learning algorithms are increasingly computationally dema...

Please sign up or login with your details

Forgot password? Click here to reset