PED-ANOVA: Efficiently Quantifying Hyperparameter Importance in Arbitrary Subspaces

by   Shuhei Watanabe, et al.
University of Freiburg

The recent rise in popularity of Hyperparameter Optimization (HPO) for deep learning has highlighted the role that good hyperparameter (HP) space design can play in training strong models. In turn, designing a good HP space is critically dependent on understanding the role of different HPs. This motivates research on HP Importance (HPI), e.g., with the popular method of functional ANOVA (f-ANOVA). However, the original f-ANOVA formulation is inapplicable to the subspaces most relevant to algorithm designers, such as those defined by top performance. To overcome this issue, we derive a novel formulation of f-ANOVA for arbitrary subspaces and propose an algorithm that uses Pearson divergence (PED) to enable a closed-form calculation of HPI. We demonstrate that this new algorithm, dubbed PED-ANOVA, is able to successfully identify important HPs in different subspaces while also being extremely computationally efficient.


page 1

page 2

page 11

page 18


Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity

We propose a new algorithm for hyperparameter selection in machine learn...

Filtrated Algebraic Subspace Clustering

Subspace clustering is the problem of clustering data that lie close to ...

Hyperparameter Optimization via Sequential Uniform Designs

Hyperparameter tuning or optimization plays a central role in the automa...

Hyperparameter Importance for Machine Learning Algorithms

Hyperparameter plays an essential role in the fitting of supervised mach...

PHS: A Toolbox for Parellel Hyperparameter Search

We introduce an open source python framework named PHS - Parallel Hyperp...

Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers

Automated hyperparameter optimization (HPO) has gained great popularity ...

Fast transforms over finite fields of characteristic two

An additive fast Fourier transform over a finite field of characteristic...

Please sign up or login with your details

Forgot password? Click here to reset