While Tuning is Good, No Tuner is Best

07/29/2018
by   Huy Tu, et al.
0

Hyperparameter tuning is the black art of automatically finding a good combination of control parameters for a data miner. While widely applied in Software Engineering, there has not been much discussion on which hyperparameter tuner is best for software analytics. To address this gap in the literature, this paper applied a range of hyperparameter optimizers (grid search, differential evolution, random search, SMAC) to defect prediction. No hyperparameter optimizer was observed to be "best" and, for one of the two evaluation measures studied here (F-measure), hyperparameter optimization, in 50% cases, was no better than using default configurations. We conclude that hyperparameter optimization is more nuanced than previously believed. While such optimization can certainly lead to large improvements in the performance of classifiers used in software analytics, it remains to be seen which specific optimizers should be endorsed.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/29/2018

Is One Hyperparameter Optimizer Enough?

Hyperparameter tuning is the black art of automatically finding a good c...
04/28/2018

Hyperparameter Optimization for Effort Estimation

Software analytics has been widely used in software engineering for many...
08/09/2020

Improving Deep Learning for Defect Prediction (using the GHOST Hyperparameter Optimizer)

There has been much recent interest in the application of deep learning ...
02/05/2019

How to "DODGE" Complex Software Analytics?

AI software is still software. Software engineers need better tools to m...
11/20/2020

Hyperparameter Optimization for AST Differencing

Computing the differences between two versions of the same program is an...
11/29/2022

Fast Hyperparameter Tuning for Ising Machines

In this paper, we propose a novel technique to accelerate Ising machines...
09/08/2016

Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?

Context: One of the black arts of data mining is learning the magic para...

Please sign up or login with your details

Forgot password? Click here to reset