Stochastic Hyperparameter Optimization through Hypernetworks

by   Jonathan Lorraine, et al.

Machine learning models are often tuned by nesting optimization of model weights inside the optimization of hyperparameters. We give a method to collapse this nested optimization into joint stochastic optimization of weights and hyperparameters. Our process trains a neural network to output approximately optimal weights as a function of hyperparameters. We show that our technique converges to locally optimal weights and hyperparameters for sufficiently large hypernetworks. We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.


page 1

page 2

page 3

page 4


Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Machine learning methods are being increasingly used in most technical a...

Deep Ranking Ensembles for Hyperparameter Optimization

Automatically optimizing the hyperparameters of Machine Learning algorit...

Local Energy Distribution Based Hyperparameter Determination for Stochastic Simulated Annealing

This paper presents a local energy distribution based hyperparameter det...

Goal-Oriented Sensitivity Analysis of Hyperparameters in Deep Learning

Tackling new machine learning problems with neural networks always means...

Two-Tailed Averaging: Anytime Adaptive Once-in-a-while Optimal Iterate Averaging for Stochastic Optimization

Tail averaging improves on Polyak averaging's non-asymptotic behaviour b...

Word2Vec applied to Recommendation: Hyperparameters Matter

Skip-gram with negative sampling, a popular variant of Word2vec original...

HyperSTAR: Task-Aware Hyperparameters for Deep Networks

While deep neural networks excel in solving visual recognition tasks, th...

Code Repositories


Code for Stochastic Hyperparameter Optimization through Hypernetworks

view repo

Please sign up or login with your details

Forgot password? Click here to reset