Tunability: Importance of Hyperparameters of Machine Learning Algorithms

by   Philipp Probst, et al.

Modern machine learning algorithms for classification or regression such as gradient boosting, random forest and neural networks involve a number of parameters that have to be fixed before running them. Such parameters are commonly denoted as hyperparameters in machine learning, a terminology we also adopt here. The term tuning parameter is also frequently used to denote parameters that should be carefully tuned, i.e. optimized with respect to performance. The users of these algorithms can use defaults of these hyperparameters that are specified in the employed software package, set them to alternative specific values or use a tuning strategy to choose them appropriately for the specific dataset at hand. In this context, we define tunability as the amount of performance gain that can be achieved by setting the considered hyperparameter to the best possible value instead of the default value. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform (Vanschoren et al., 2013) using six of the most common machine learning algorithms for classification and regression and apply our measures to assess the tunability of their parameters. The results yield interesting insights into the investigated hyperparameters that in some cases allow general conclusions on their tunability. Our results may help users of the algorithms to decide whether it is worth to conduct a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.


page 1

page 2

page 3

page 4


Importance of Tuning Hyperparameters of Machine Learning Algorithms

The performance of many machine learning algorithms depends on their hyp...

To tune or not to tune? An Approach for Recommending Important Hyperparameters

Novel technologies in automated machine learning ease the complexity of ...

MindOpt Tuner: Boost the Performance of Numerical Software by Automatic Parameter Tuning

Numerical software is usually shipped with built-in hyperparameters. By ...

High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms

Hyperparameters in machine learning (ML) have received a fair amount of ...

Gradient-based Quadratic Multiform Separation

Classification as a supervised learning concept is an important content ...

Intrinsic uncertainties and where to find them

We introduce a framework for uncertainty estimation that both describes ...

Hyperparameters and Tuning Strategies for Random Forest

The random forest algorithm (RF) has several hyperparameters that have t...

Please sign up or login with your details

Forgot password? Click here to reset