The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection

11/09/2021
by   Shubhankar Mohapatra, et al.
Google
University of Waterloo
MIT
0

Hyperparameter optimization is a ubiquitous challenge in machine learning, and the performance of a trained model depends crucially upon their effective selection. While a rich set of tools exist for this purpose, there are currently no practical hyperparameter selection methods under the constraint of differential privacy (DP). We study honest hyperparameter selection for differentially private machine learning, in which the process of hyperparameter tuning is accounted for in the overall privacy budget. To this end, we i) show that standard composition tools outperform more advanced techniques in many settings, ii) empirically and theoretically demonstrate an intrinsic connection between the learning rate and clipping norm hyperparameters, iii) show that adaptive optimizers like DPAdam enjoy a significant advantage in the process of honest hyperparameter tuning, and iv) draw upon novel limiting behaviour of Adam in the DP setting to design a new and more efficient optimizer.

READ FULL TEXT

page 8

page 9

page 18

01/27/2023

Practical Differentially Private Hyperparameter Tuning with Subsampling

Tuning all the hyperparameters of differentially private (DP) machine le...
11/03/2022

Revisiting Hyperparameter Tuning with Differential Privacy

Hyperparameter tuning is a common practice in the application of machine...
06/09/2023

DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework

Hyperparameter optimization, also known as hyperparameter tuning, is a w...
08/09/2021

Efficient Hyperparameter Optimization for Differentially Private Deep Learning

Tuning the hyperparameters in the differentially private stochastic grad...
10/25/2019

On the Tunability of Optimizers in Deep Learning

There is no consensus yet on the question whether adaptive gradient meth...
03/30/2022

Adaptive Private-K-Selection with Adaptive K and Application to Multi-label PATE

We provide an end-to-end Renyi DP based-framework for differentially pri...
01/16/2022

Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters

Machine learning methods are being increasingly used in most technical a...

Please sign up or login with your details

Forgot password? Click here to reset