While Tuning is Good, No Tuner is Best

by   Huy Tu, et al.

Hyperparameter tuning is the black art of automatically finding a good combination of control parameters for a data miner. While widely applied in Software Engineering, there has not been much discussion on which hyperparameter tuner is best for software analytics. To address this gap in the literature, this paper applied a range of hyperparameter optimizers (grid search, differential evolution, random search, SMAC) to defect prediction. No hyperparameter optimizer was observed to be "best" and, for one of the two evaluation measures studied here (F-measure), hyperparameter optimization, in 50% cases, was no better than using default configurations. We conclude that hyperparameter optimization is more nuanced than previously believed. While such optimization can certainly lead to large improvements in the performance of classifiers used in software analytics, it remains to be seen which specific optimizers should be endorsed.


page 1

page 2

page 3

page 4


Is One Hyperparameter Optimizer Enough?

Hyperparameter tuning is the black art of automatically finding a good c...

Hyperparameter Optimization for Effort Estimation

Software analytics has been widely used in software engineering for many...

Improving Deep Learning for Defect Prediction (using the GHOST Hyperparameter Optimizer)

There has been much recent interest in the application of deep learning ...

How to "DODGE" Complex Software Analytics?

AI software is still software. Software engineers need better tools to m...

Hyperparameter Optimization for AST Differencing

Computing the differences between two versions of the same program is an...

Fast Hyperparameter Tuning for Ising Machines

In this paper, we propose a novel technique to accelerate Ising machines...

Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?

Context: One of the black arts of data mining is learning the magic para...

Please sign up or login with your details

Forgot password? Click here to reset